xpra icon
Bug tracker and wiki

Version 18 (modified by Antoine Martin, 6 years ago) (diff)

make it easier to read?

Debugging


Some areas of the system may have dedicated pages which may be useful for debugging, in combination with the information contained here:

First Things

Always try to narrow it down as much as possible, by turning off as many features as possible (clipboard, etc) and trying various encodings.

Debug Builds

If a debug build is available, please use it. It may provide more useful diagnostic messages. You can generate debug builds by calling setup.py with the --with-debug flag (or by setting DEBUG=1 in the MS Windows BAT file)



Via Logging

Sometimes the problem is so obvious that you will simply get the error directly from the command line or in the server log file. You may also want to run the server with the --no-daemon switch to more easily keep an eye on its log file and status.

Otherwise, a good first step is to start xpra from the command line and add "-d all" to it. The amount of data logged can be overwhelming, so make sure you log it or redirect it to a file that you can then grep to find and extract the data you are looking for.


Some modules do not log by default, either to reduce the amount of logging or to avoid the overhead of logging in critical paths. Logging can generally be enabled using environment variables, you can find the list of such runtime configuration options with:

egrep -re "XPRA_.*DEBUG|XPRA_.*LOG" src/xpra | grep 'os.environ'
egrep -re "XPRA_.*DEBUG|XPRA_.*LOG" src/wimpiggy | grep 'os.environ'

At time of writing this shows the following options:

  • XPRA_KEYBOARD_DEBUG
  • XPRA_DAMAGE_DEBUG (damage events show which regions have been repainted)
  • XPRA_DEBUG_SOUND
  • XPRA_X11_LOG (enables logging for X11 events)
  • XPRA_X11_DEBUG (logs all X11 events through going through our loop - this is very verbose)

Other Environment Variables

There are other environment variables which can be used to tune the behaviour of the system and may be used for debugging or testing, you can obtain the full list with:

grep 'os.environ' src/xpra
grep 'os.environ' src/wimpiggy

Notably:

  • some x264 attributes and thresholds: XPRA_X264_*_PROFILE, XPRA_X264_*_MIN_QUALITY, XPRA_X264_*_QUALITY
  • lossless damage threshold values (number of pixels): XPRA_MAX_NONVIDEO_PIXELS and MAX_NONVIDEO_OR_INITIAL_PIXELS
  • sound test (use fake sound source on/off): XPRA_SOUND_TEST
  • opengl (on/off): XPRA_OPENGL
  • use cython maths (on/off): XPRA_CYTHON_MATH
  • dump latency debug information to log file (on/off): XPRA_DEBUG_LATENCY
  • try to schedule other threads in network and damage loops (on/off - see #181): XPRA_YIELD
  • check that we obey our network call threading rules (on/off): XPRA_CHECK_THREAD
  • use PIL for parsing png and jpeg packets (on/off): XPRA_USE_PIL

Please refer to the actual code for details.



Network Traffic

Seeing what is being exchanged between the client and server can be useful at times, an easy way to achieve this is to use tcp mode (ie: --bind-tcp=0.0.0.0:10000 option) without packet compression (-z 0 option) then the packets can be looked at with your favourite packet inspection tool, ie:

ngrep -p 10000

The packet type is the first thing in each packet and it is a simple string which makes it easy to observe traffic. With version 0.9 onwards, it is best to set XPRA_USE_ALIASES=0 to ensure that the packets will use plain-text headers rather than numeric aliases.



GDB

When dealing with crashes ("core dumped"), the best way to debug is to fire gdb.

Getting a Backtrace

xpra start ...
# or xpra attach ...
ps -ef | grep xpra
gdb python $PID_OF_XPRA_PROCESS_TO_DEBUG
# wait for it to load all the debug symbols
(gdb) continue

Then once you get a crash, gdb should show you its prompt again and you can extract the python stacktrace with py-bt and the full stacktrace with bt. Having both is useful. Note: installing the required "debug" symbol packages for your distribution is out of scope, please refer to your vendor's package manager for details (ie: debian and yum debuginfo-install).

X11 errors

Occasionally you may encounter an X11 crash (BadDrawable, BadWindow, etc). These are more tricky. You may need to set a breakpoint:

(gdb) break gdk_x_error

(or _XError if gdk_x_error is not found) And get a backtrace from there when gdb hits it.

Another useful environment variable is XPRA_X11_DEBUG_EVENTS, it allows you to specify a CSV list of X11 events to log specifically, ie:

XPRA_X11_DEBUG_EVENTS="PropertyNotify,MotionNotify" xpra ...

The full list of event names can be obtained by using an invalid value, which will trigger a warning message, or by looking at the source.

Unfortunately, because of the asynchronous nature of X11 calls, the error may generate this crash some time after the event that caused it. In this case, we may want to force all X11 calls to be synchronized (which will hurt performance and may even hide the bugs - beware of heisenbugs!), trace all X11 calls, etc. this modified error.py allows you to do just that (see the constants at the top).


Venerable Print Statements

When all else fails, or just when appropriate, sprinkling some print statements around the critical sections of code is often the best way to get a clearer picture of what is really going on..


Memory Leaks

In C code

If the memory leak is one of the Cython parts or in a C library, use regular tools, like valgrind. When using valgrind, the connection timeout may be exceeded because the client will be slowed down. Increasing this timeout will be necessary to be able to use the client with Valgrind.

In python code

Needs documenting, objgraph is a good place to start.


Debugging in Windows

Using Microsoft Visual Studio Express

You will need to create a new, empty project (File->New->Project), select "General" and "Empty Project". Visual Studio will require a name for the project and will ask for a place to save it. This project will contain nothing but basic information to start Xpra with the debugger, so save it somewhere for later use.

You will be placed in the tree, containing the "Solution" and the project you've created. Right click on the project in the tree and select "properties".

Go to "Debugging", in fill in the fields to attach your Xpra client - example below.

No image "screenshot_1363196248.jpg" attached to Debugging

Then you can do F5 to start the application with the debugger.

Alternatively, if you set "Attach" to yes, the debugger will ask you to what running process you want to attach upon pressing F5.

Attachments (1)

  • error.py (6.7 KB) - added by Antoine Martin 6 years ago. debug version of error.py which allows us to force sync calls, add logging, etc

Download all attachments as: .zip