xpra icon
Bug tracker and wiki

http://xpra.org/icons/bugs.png

Debugging

For capturing the data required for filing bug reports, please see Bug Report Tool, wiki/SessionInfo is also useful for diagnostics.


Some areas of the system may have dedicated pages which may be useful for debugging, in combination with the information contained here:


First Steps

Always try to narrow it down as much as possible, by turning off as many features as possible (clipboard, etc) and trying various encodings. Try another operating system, try a different version, etc

xpra info is always good to have.

Debug Logging is the most commonly used debugging technique, enable logging for the subsystem categories relevant to your problem.

Debug Builds

For application crashes, please try to use a debug build if one is available or install the corresponding debuginfo packages. It may provide more useful diagnostic messages.

You can generate debug builds by calling setup.py with the --with-debug flag.

GDB

When dealing with crashes ("core dumped"), the best way to debug is to fire gdb to get a backtrace.

Attaching to an existing xpra process

Find the pid of the xpra process:

ps -ef | grep xpra

Then:

gdb python $PID_OF_XPRA_PROCESS_TO_DEBUG
# wait for it to load all the debug symbols
(gdb) continue

Starting xpra in gdb

gdb python
run /usr/bin/xpra start ...

or just: gdb --args /usr/bin/python /usr/bin/xpra start ...

Getting the backtrace

Then once you get a crash, gdb should show you its prompt again and you can extract the python stacktrace with py-bt and the full stacktrace with bt. Having both is useful.

Note: installing the required "debug" symbol packages for your distribution is out of scope, please refer to your vendor's package manager for details (ie: debian and yum debuginfo-install).


More advanced techniques below:

Other Environment Variables

There are other environment variables which can be used to tune the behaviour of the system and may be used for debugging or testing, you can obtain the full list with:

grep 'os.environ' src/xpra

Some examples:

  • some x264 attributes and thresholds: XPRA_X264_*_PROFILE, XPRA_X264_*_MIN_QUALITY, XPRA_X264_*_QUALITY
  • lossless damage threshold values (number of pixels): XPRA_MAX_NONVIDEO_PIXELS
  • sound test (use fake sound source on/off): XPRA_SOUND_TEST
  • use cython maths (on/off): XPRA_CYTHON_MATH
  • try to schedule other threads in network loop (on/off - see #181): XPRA_YIELD
  • use PIL for parsing png and jpeg packets (on/off): XPRA_USE_PIL

Please refer to the actual code for details.

Fault Injection

You can also inject faults in various places to trigger hard to reproduce error conditions.

For more details see #910.

Network Traffic

Seeing what is being exchanged between the client and server can be useful at times, an easy way to achieve this is to use tcp mode (ie: --bind-tcp=0.0.0.0:10000 option) without packet compression (-z 0 option) then the packets can be looked at with your favourite packet inspection tool, ie:

ngrep -p 10000

The packet type is the first thing in each packet and it is a simple string which makes it easy to observe traffic. With version 0.9 onwards, it is best to set XPRA_USE_ALIASES=0 to ensure that the packets will use plain-text headers rather than numeric aliases.

xtrace

(X11 traffic)

xtrace allows us to monitor the traffic between the xpra server and the Xvfb (X11 server), or between the client and its display server (Posix only).

  • server setup: because the server normally launches its own xvfb, we need to tell it to use an existing one and we interpose xtrace in between:
    • start your xvfb:
      Xvfb ... :10
      
    • start xtrace forwarding from :10 to :11:
      xtrace -k -d :10 -D :11
      
    • start an xpra server on the existing :11 display:
      xpra start --use-display :11
      


  • client setup: simply run the client via xtrace:
    xtrace xpra attach ..
    

X11 errors

Occasionally you may encounter an X11 crash (BadDrawable, BadWindow, etc). These are more tricky. You may need to set a breakpoint:

(gdb) break gdk_x_error

(or _XError if gdk_x_error is not found) And get a backtrace from there when gdb hits it.

Another useful environment variable is XPRA_X11_DEBUG_EVENTS, it allows you to specify a CSV list of X11 events to log specifically, ie:

XPRA_X11_DEBUG_EVENTS="PropertyNotify,MotionNotify" xpra ...

The full list of event names can be obtained by using an invalid value, which will trigger a warning message, or by looking at the source.

Unfortunately, because of the asynchronous nature of X11 calls, the error may generate this crash some time after the event that caused it. In this case, we may want to force all X11 calls to be synchronized (which will hurt performance and may even hide the bugs - beware of heisenbugs!), trace all X11 calls, etc. this modified error.py allows you to do just that (see the constants at the top).

Venerable Print Statements

When all else fails, or just when appropriate, sprinkling some print statements around the critical sections of code is often the best way to get a clearer picture of what is really going on..

Memory Leaks

In X11 Server

xrestop is a good way of quickly visualising resource usage.

In C code

If the memory leak is one of the Cython parts or in a C library, use regular tools, like valgrind. When using valgrind, the connection timeout may be exceeded because the client will be slowed down. Increasing this timeout will be necessary to be able to use the client with Valgrind. With r4861 onwards, you can do this using the XPRA_SOCKET_TIMEOUT env var:

XPRA_SOCKET_TIMEOUT=30 xpra attach ...

In python code

As of version 0.13, it is possible to get the server to print every 10 seconds the objects which seem to be leaking by starting the server with the environment variable XPRA_DETECT_LEAKS set to 1.


Needs documenting better..

Last modified 4 months ago Last modified on 01/12/17 12:24:36

Attachments (1)

  • error.py (6.7 KB) - added by Antoine Martin 4 years ago. debug version of error.py which allows us to force sync calls, add logging, etc

Download all attachments as: .zip