xpra icon
Bug tracker and wiki

This bug tracker and wiki are being discontinued
please use https://github.com/Xpra-org/xpra instead.


Opened 8 months ago

Last modified 6 months ago

#2967 assigned task

migrate away from trac and svn

Reported by: Antoine Martin Owned by: Antoine Martin
Priority: major Milestone: 4.1
Component: external Version: 3.0.x
Keywords: Cc:

Description (last modified by Antoine Martin)

They have served us well for over 10 years, but without any meaningful progress, especially with trac still not supporting python3 properly..
It's time to move on.


Options (in no particular order):


There are pros and cons for each option:

  • amount of work for migrating
  • amount of maintenance work
  • cost
  • accessibility
  • making it easier for others to contribute
  • maintaining existing documentation links and resources (could keep trac in read-only mode for a while)
  • preserving code authors in git history
  • ci / cd
  • use oauth if self-hosted?

Attachments (3)

TracWiki2MediaWiki.pl (36.6 KB) - added by Antoine Martin 6 months ago.
perl script found here: https://www.mediawiki.org/w/index.php?oldid=2103135#Code
extract-trac-attachments.sh (1.4 KB) - added by Antoine Martin 6 months ago.
needed to keep links to the attachments working
trac-hub.patch (3.0 KB) - added by Antoine Martin 6 months ago.
remove email addresses from tickets, remove dependency on unused mysql backend, increase delay to prevent hitting the github API rate limits, don't post empty comments, fix wiki links

Download all attachments as: .zip

Change History (9)

comment:1 Changed 8 months ago by Antoine Martin

Description: modified (diff)
Status: newassigned

As an aside, if going for the hosted solution, it might be worth hosting the mailing list too. (there are many mailman hosting services out there, all relatively cheap)

See also: move to new signing keys (#2968)

It would be nice if some of the release script could preserve the incremental revision number feature of svn:

Last edited 6 months ago by Antoine Martin (previous) (diff)

comment:2 Changed 8 months ago by stdedos

I would much more prefer Github than Gitlab, but that may come down purely on personal preference.

There are tidbits here and there that make everything feel more palpable (e.g. the URLs are very clearly read and manipulated to move from here to there in Github), and there are no "weird behaviors (e.g. the annoying @-reference automatically assigning the issue to the referencee as to-do that exists in Gitlab)

For git-specific stuff, I guess you have already seen https://git-scm.com/book/en/v2/Git-and-Other-Systems-Migrating-to-Git (and might be a good idea to update it if/after you have improved something).

Last edited 8 months ago by stdedos (previous) (diff)

comment:3 Changed 7 months ago by brief

If considered that this step will be necessary again in the future, data should be available for migration.

Self hosted solutions (e.g. gitlab) guarantee to be able to access this data.

comment:4 Changed 6 months ago by Antoine Martin

  • generate new ssh and gpg keys for antoine@xpra.org and xpra@xpra.org (see #2968)
  • enable 2factor yubikey with github
  • create a user mapping authors.txt from known svn users
  • convert using svn2git:
    mkdir Xpra
    cd Xpra
    # use svn tags as git branches:
    svn2git https://xpra.org/svn/Xpra \
        --trunk trunk \
        --branches tags --notags \
        --metadata \
        --exclude web \
        --exclude trunk/fakexinerama \
        --exclude trunk/scripts/fakeXinerama-snapshot.sh \
        --authors ~/authors.txt
    # then remove EOL branches (too many):
    for x in v0.0.7.x v0.1.x v0.10.x v0.11.x v0.12.x v0.13.x v0.14.x v0.15.x v0.16.x v0.17.x v0.2.x v0.3.x v0.4.x v0.5.x v0.6.x v0.7.x v0.8.x v0.9.x v2.0.x v2.1.x v2.2.x v2.3.x v2.4.x v2.5.x; do git branch -D $x; done
    # create tags for specific versions (the last one from each supported branch only):
    for V in v1.0.14 v3.0.13 v4.0.6; do
        echo $V
        MV=`echo $V | cut -c2-4`
        git checkout "v"$MV".x"
        NOV=`echo $V | cut -c2-10`
        #get the xpra source snapshot:
        XPRAV="xpra-$NOV"
        wget "https://xpra.org/src/$XPRAV.tar.xz"
        tar -Jxvf $XPRAV.tar.xz $XPRAV/svn-version
        #get the svnversion from it:
        SVNVERSION=`cat $XPRAV/svn-version | xargs echo -n`
        rm -fr $XPRAV*
        echo "r$SVNVERSION"
        #find the git commit for this svn version number:
        COMMIT=`git log --all --grep="$SVNVERSION" | grep "^commit" | awk '{print $2}'`
        echo "git-commit: $COMMIT"
        #preserve the date:
        GIT_COMMITTER_DATE=`git show --format=%aD $COMMIT | head -n 1 | xargs echo -n`
        #tag it:
        git tag $V $COMMIT
        unset GIT_COMMITTER_DATE
    done
    # configure email and keys:
    git config user.email "antoine@xpra.org"
    git config user.signingkey 0xDCDCA2B6F88896E4
    # "upload" to github:
    git remote add origin git@github.com:Xpra-org/xpra.git
    git push --set-upstream origin master
    git push --all origin
    git push --tags origin
    
  • move the debian, rpmbuild, osx and win32 directories to one sub-directory? (win32 will need more work as it contains scripts used by setup.py)
  • move the code one level up: remove redundant src dir
  • split the HTML5 client - from the html5 folder directly because I couldn't make svn2git honour --exclude (see -exclude parameter doesn't seem to skip folders as directed):
    $ svn2git https://xpra.org/svn/Xpra \
        --trunk trunk/src/html5 \
        --nobranches \
        --notags \
        --metadata \
        --authors ~/authors.txt \
        --revision 5070
    # configure email and keys:
    git config user.email "antoine@xpra.org"
    git config user.signingkey 0xDCDCA2B6F88896E4
    # "upload" to github:
    git remote add origin git@github.com:Xpra-org/xpra-html5.git
    git push --set-upstream origin master
    git push --all origin
    git push --tags origin
    
  • convert trac tickets to github issues:
    # generate map from revision to commit:
    git log  | egrep "git-svn-id:|^commit" | awk '/    git-svn-id:/{print$0p}{p=$0}' | sed 's+.*trunk@++g' | awk '{print $1" "$3}' > rev-map-file
    # edit config: add token, etc - then:
    bundle exec trac-hub -r ./rev-map-file
    

Still in progress:

  • git2svn generates URLs like https://xpra.org/svn/Xpra/trunk@28364, apache will need to rewrite those to valid URLs
  • configure github to never generate pull requests from stable branches to master? can it be done or do we have to just ignore it?
  • make it possible to generate the releases directly on github?
  • add README, LICENSE, etc
  • make the repository public
  • migrate the wiki pages (requires a public repository)
  • edit trac and www to point to github
  • lock svn and trac
  • make svn and trac backups public
  • send announcements
  • transfer relevant issues to the xpra-html5 repository

etc

Last edited 6 months ago by Antoine Martin (previous) (diff)

Changed 6 months ago by Antoine Martin

Attachment: TracWiki2MediaWiki.pl added

Changed 6 months ago by Antoine Martin

Attachment: extract-trac-attachments.sh added

needed to keep links to the attachments working

Changed 6 months ago by Antoine Martin

Attachment: trac-hub.patch added

remove email addresses from tickets, remove dependency on unused mysql backend, increase delay to prevent hitting the github API rate limits, don't post empty comments, fix wiki links

comment:5 Changed 6 months ago by Antoine Martin

  • trac ticket migration much improved with the patch above:
    • removed email addresses from tickets as users did not agree to have their email address made public
    • remove dependency on unused mysql backend, we only need the sqlite backend with this installation - and for whatever reason, installation of the mysql backend would fail on this gentoo system
    • increase delay between polling requests to prevent hitting the github API rate limits
    • don't post empty comments
    • fix wiki links (my really ugly ruby code..)
  • apache redirect
    	RewriteRule "^/svn/Xpra/(.*)@(.*)$" "/svn/Xpra/$1?p=$2" [R]
    
  • remove ability to create tickets in trac using trac-admin:
    permission remove authenticated  TICKET_APPEND
    permission remove authenticated  TICKET_CREATE
    permission remove authenticated  TICKET_MODIFY
    permission remove authenticated  WIKI_CREATE
    permission remove authenticated  WIKI_MODIFY
    

The repositories and tickets are now available here: https://github.com/Xpra-org

comment:6 Changed 6 months ago by migration script

this ticket has been moved to: https://github.com/Xpra-org/xpra/issues/2967

Note: See TracTickets for help on using tickets.