Announce: SvnBridge: new version (strongly improved)

Oct 30, 2015 at 3:46 PM
Hello all,

I implemented many updates
to the unfortunately relatively unstable SvnBridge project
(which tries to achieve an admittedly somewhat complex transversion of SCM protocols).

My changes (around 900 commits) to the upstream project's history
can be found at:

github.com/andim2/SvnBridge

(current version provided in "preview" branch, since I consider it to be in "preview" state,
waiting to gather reviewer comments and getting further fixup etc.)


=== ChangeLog (mostly lists notable features only) ===

This work provides many fixes, improvements and features
(many listed below, but see commits for further details):

. support for newer TFS201x web service protocol
(via TfsLibrary-related updates compatibly adopted from "weird" svnbridge2 fork commit)
. support for newer TFS201x Collection URL syntax
(desktop SvnBridge code path necessity only?)
. many severe memory management (garbage collection) lifetime issues fixed
. improperly non-incrementally streamy operation implementation
(very annoying accumulation of multi-GB huge-blob data)
eradicated as much as possible at the moment
. filesystem item case insensitivity issues near-fully fixed, on both checkout and commit side
(this includes very invasive low-level filtering of completely b0rken TFS-side changeset database content!!)
. severe issues in update diff calculator corrected (total calculation over certain commit diff ranges, reverse updating etc.)
. unfortunately there are some update diff calculation issues known to remain still...
. fix svn blame (in general, and when hitting deleted files in history)
. huge amount of transcoding bugs / weaknesses fixed
(full Unicode character space of SCM items now pretty reliably working)
. improved support of storage mechanism of WebDAV properties ([INCOMPATIBLE] corrected file naming, severely buggy props storage improved, fixed layer violations)
(WARNING: property storage layer remains semi-working / problematic, thus I added default-disabled read/write config settings to avoid using it - subsequent commit failure may occur when having stored properties)
. WebDAV support now working pretty nicely
(Cadaver, Konqueror webdav://)
. support Non-NULL SVN UpdateTarget
. SVN COPY previous-item support (#15317)
. add missing support of various /!svn/foo paths at various request handlers
. support required HTTP/1.1 chunked-transfer encoding (unbreaks newer TortoiseSVN)
. support HTTP Keep-Alive, HTTP Content-Encoding ([gzip], deflate)
. enable TCP Keep-Alive
. AsyncItemLoader (crawler worker):
. tons of fixes/improvements (fix race conditions, fully asynchronous and multi-parallel operation, etc.)
. but: no retries yet in case of download fail
(may happen relatively easily e.g. when trying to fetch big files from TFS)
. enabling of non-localhost interface bind
(to go beyond same-desktop-only deployments of desktop SvnBridge)


=== Misc. notes ===

. global stability/usability assessment:
a ton better than the roughly pre-beta version which has been offered -
I'd say it's nearing "semi-stable", however several issues are certain to come up still
. has been used successfully to fetch several (sometimes very) large repositories
. ATTENTION WARNING (truly or potentially) crucially deployment-incompatible items:
. props storage (now using more compatible / descriptive .sbsvnprops / .sbsvnpropsfile)
. work item xml files template variable changed (make sure to deploy our updated files)
. properly supported full case sensitivity (hopefully not problematic)
. several man-months of work
. diffstat (encompassing pretty much only precisely restricted, isolated changes):
167 files changed, 19568 insertions(+), 4246 deletions(-)
. Performance: I haven't done any specific before/after tests
(these aren't easily possible anyway given the several critical infrastructure fixes),
but due to things such as
many implementation optimizations, caching, persistent connections, compression,
it actually ought to be much faster than before
despite the excessive overhead which is required to sanitize incredible TFS bugs
. however, conversion complexity still remains huge at certain spots (huge commits may take a looooong time, risking hitting timeouts of e.g. TortoiseSVN or svn where configuration such as ~/.subversion/servers http-timeout has not been adapted)
. since these are many, many changes and I have tested them with specific repos only,
you are advised to be careful when deploying / testing this project (especially this initial early public review)
. so far in recent times I have tested Desktop SvnBridge only - IOW WARNING: SvnBridge-Via-IIS has not been tested recently and may easily be broken!
. I decided to release this now since it's about time to have it reach some wider audience (violated "RERO" principle)
. I tried to ensure continued successful passing of (almost...) all unit tests
. added tests for some of the bugs that I fixed
(and added unit testing README, at Tests/README_start_here.txt)
. did several corrections due to FxCop analysis
. fully build-bisectable commit history
. note that while I did test subversion command line and TortoiseSVN,
working with subversion-only clients may be rather painful:
since svn is not distributed version control (DVCS),
it keeps needing to go back to the server for history etc.,
which is very (sometimes awfully) slow
due to the massive TFS <-> SVN protocol conversion effort overhead -
thus better use something like git-svn instead
. I decided to do a lot of work on SvnBridge rather than the various git TFS adapters
since IMHO it's better to have a very accessible SVN-based base protocol as a "lingua franca"
(which could then be used by many SCM clients, e.g. git-svn etc.)
. if working with TFS remains "challenging" despite improved SCM adapters,
then it might be worthwhile to use truly cross-platform (server, clients, development) SCM tools
such as git, svn or Mercurial instead
(these may have benefits such as:
. properly case-sensitive case-preserving handling of filesystem item space [!!]
. supporting file properties / permissions
. supporting externals / sub module
. avoiding branch-by-copy overhead
. avoiding a painful Windows dependency [!!]
. better user interfaces even
)
. interesting items which could be worth revisiting (some will/may be fixed) and/or reporting:
http://svnbridge.codeplex.com/discussions/315884 (https)
http://svnbridge.codeplex.com/discussions/256258 (fully unicode aware support of paths, now supported)
http://svnbridge.codeplex.com/discussions/258924 (branch Merge, now supported)



I was lucky enough to have been allowed to publish the changes to this public Internet project
which I did during my work time (thank you!!).


I would be very grateful for any feedback (issues, improvements etc.),
preferably as a public reply (e.g. here).


HTH & HAND,

Andreas Mohr
Dec 1, 2015 at 8:31 PM
An installation on IIS (8.0, I've been told) of this HEAD revision seems to work fine after some initial testing.


However, diff engine issues are somewhat more involved (leading to repo fetch issues in case of more complex Changesets containing Changes such as Rename|Undelete), and I'm currently working on getting this fixed in a hopefully robust manner. Stay tuned!
Mar 14, 2016 at 3:53 PM
A have using TFS 2013 project and there is problem with SvnBridge.
After compiled SvnBridge new version I can't find client exec.
There is also some dll with server ( SvnBridgeServer, CodePlex.TfsLibrary, SvnBridge.Library).

Can you provide some documentation how to install new version of SvnBridge?
Mar 16, 2016 at 2:54 PM
lukaszp1985 wrote:
A have using TFS 2013 project and there is problem with SvnBridge.
I am using TFS2013 here as well.
After compiled SvnBridge new version I can't find client exec.
Hmm, are you sure that build worked correctly without any compile errors etc.?
(if not, pray tell how exactly your build fails. That might be interesting...)
Also, how are you building it? Most often I'm running it directly in MSVS (2010),
i.e. not via external builds e.g. via msbuild.

Once build ended up successful, there should be a binary called SvnBridge.exe provided within [SvnBridge_working_copy_root]/SvnBridge/bin/[Build_Configuration_Name]

That SvnBridge.exe binary can then be launched directly, for use by the local desktop (SvnBridge desktop mode).
In the case of my extended SvnBridge version, when extending network interface bind to non-localhost configuration (see file app.config , and source [SvnBridge_working_copy_root]/SvnBridgeLibrary/Infrastructure/Configuration.cs ), that SvnBridge instance can be used for any client beyond that local Windows desktop, probably sufficiently securely (albeit cleartext!) authenticated (but there is no security guarantee either, in addition to a lack of general usability guarantee of this SvnBridge implementation!).
There is also some dll with server ( SvnBridgeServer, CodePlex.TfsLibrary, SvnBridge.Library).
Yes, most of these are required in most circumstances, and when doing deployment on IIS, these will need to be copied there, too (in addition to potentially custom-modified required .xml template files).
Can you provide some documentation how to install new version of SvnBridge?
I'm afraid for the moment that will need to do it (in an ad-hoc manner), however if something is missing then I'll extend stuff.
And longer-term I'm planning to provide a sufficiently detailed central markdown README.


Note that I'm currently fixing another annoying issue (commit-side DAV COPY / WRITE / DELETE requests cannot be transformed into TFS Pending Change:s Rename, Edit at least on newer TFS, due to some TFS consistency checks rightfully barfing during attempted transformation of Pending Change:s).
Apart from this, my current fixed local version is working almost fully satisfactorily, with few known annoying bugs remaining.

HTH and sorry for the slight delay!

Thank you for your report, greatly appreciated!
Mar 16, 2016 at 4:11 PM
I am building your project in VS Express 2015 for Desktop. After open solution from codeplex source I get error:
Unsupported
This version of Visual Studio is unable to open the following projects. The project types may not be installed or this version of Visual Studio may not support them. 
For more information on enabling these project types or otherwise migrating your assets, please see the details in the "Migration Report" displayed after clicking OK.
     - SvnBridgeViaIis, "D:\temp\SvnBridge-preview\SvnBridge-preview\SvnBridgeViaIis\SvnBridgeViaIis.csproj"

No changes required
These projects can be opened in Visual Studio 2015, Visual Studio 2013, Visual Studio 2012, and Visual Studio 2010 SP1 without changing them.
     - TestsUnit, "D:\temp\SvnBridge-preview\SvnBridge-preview\Tests\TestsUnit.csproj"
     - SvnBridge, "D:\temp\SvnBridge-preview\SvnBridge-preview\SvnBridge\SvnBridge.csproj"
     - TestsIntegration, "D:\temp\SvnBridge-preview\SvnBridge-preview\TestsIntegration\TestsIntegration.csproj"
     - Tools.TCPTrace, "D:\temp\SvnBridge-preview\SvnBridge-preview\TCPTrace\Tools.TCPTrace.csproj"
     - TestsProtocol, "D:\temp\SvnBridge-preview\SvnBridge-preview\TestsProtocol\TestsProtocol.csproj"
     - SvnBridge.Library, "D:\temp\SvnBridge-preview\SvnBridge-preview\SvnBridgeLibrary\SvnBridge.Library.csproj"
     - TestsEndToEnd, "D:\temp\SvnBridge-preview\SvnBridge-preview\TestsEndToEnd\TestsEndToEnd.csproj"
     - TestsRequiringTfsClient, "D:\temp\SvnBridge-preview\SvnBridge-preview\TestsRequiringTfsClient\TestsRequiringTfsClient.csproj"
     - SvnBridge.PerfCounter.Installer, "D:\temp\SvnBridge-preview\SvnBridge-preview\SvnBridge.PerfCounter.Installer\SvnBridge.PerfCounter.Installer.csproj"
     - SvnBridgeServer, "D:\temp\SvnBridge-preview\SvnBridge-preview\SvnBridgeServer\SvnBridgeServer.csproj"
     - Tools.HttpSend, "D:\temp\SvnBridge-preview\SvnBridge-preview\HttpSend\Tools.HttpSend.csproj"
     - TfsLibrary, "D:\temp\SvnBridge-preview\SvnBridge-preview\TfsLibrary\TfsLibrary.csproj"
     - SvnBridge, "D:\temp\SvnBridge-preview\SvnBridge-preview\SvnBridge.sln"
Migration Report:
SvnBridgeViaIis\SvnBridgeViaIis.csproj: The application which this project type is based on was not found. Please try this link for further information: http://go.microsoft.com/fwlink/?LinkID=299083&projecttype=349c5851-65df-11da-9384-00065b846f21
But then I can build project SvnBridge and run from bin file SvnBridge.exe. But I can't use SvnBridge desktop mode, because TFS 2013 is available on network location (via http://10.0.1.100:8080/tfs), so I need SvnBridge Server (my ip: 10.0.1.200).
After configure IIS I try to use TortoiseSVN repo-browser with URL http://10.0.1.100:8083/DefaultCollection/Project and get error: Error running context. An error occurred during authentication.
If I try open website locally (localhost:8083) I get 401.2 error (lang PL):
Translated from Polish: You do not have permission to view this page. The reason is bad authentication headers.
Moduł: IIS Web Core
Powiadomienie: AuthenticateRequest
Obsługa: SvnBridgeHandler
Kod błędu: 0x80070005
Żądany adres URL: http://localhost:8083/
Ścieżka fizyczna: :\inetpub\wwwroot\tfs_server
Metoda logowania: Jeszcze nie ustalono
Użytkownik logowania: Jeszcze nie ustalon

All authentication methods are off:
 <system.webServer>
            <security>
                <authentication>
                    <anonymousAuthentication enabled="false" />
                    <windowsAuthentication enabled="false" />
                </authentication>
            </security>
        </system.webServer>
To access to repo by VS I have to provide login and password for my domain user.

Also I have some questions:
Do I have to run Setup.reg from SvnBridgeServer directory?
What is DomainIncludesProjectName parameter from Web.config?
Which address is correctly to enter in TortoiseSVN repo-browser: http://10.0.1.100:8083/tfs/DefaultCollection/Project or without tfs - http://10.0.1.100:8083/DefaultCollection/Project?

Server with TFS info:
Windows 7 Pro
IIS 7.5.7600.16385
IP: 10.0.1.100
IIS app port: 8083

Client:
TortoiseSVN 1.9.3, Build 27038 - 64 Bit , 2015/12/13 11:53:16
IP: 10.0.1.200

Web.config:
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<appSettings>
<add key="LogPath" value="C:\inetpub\wwwroot\tfs_server\App_Data\Logs" />
<add key="LogCancelErrors" value="True" />
<add key="UseCodePlexServers" value="False" />
<add key="DomainIncludesProjectName" value="True" />    
<add key="TfsUrl" value="http://10.0.1.100:8080/tfs" />
<add key="ReadAllUserDomain" value="DOMAIN_NAME" />
<add key="ReadAllUserName" value="DOMAIN_USER" />
<add key="ReadAllUserPassword" value="PASS" />
<add key="CodePlexAnonUserDomain" value="" />
<add key="CodePlexAnonUserName" value="" />
<add key="CodePlexAnonUserPassword" value="" />
</appSettings>
<system.web>
<httpRuntime maxRequestLength="500000" /> <!-- 250+ MB -->
<customErrors mode="Off" />
<compilation debug="true" />
<authentication mode="None" />
    <identity impersonate="false" />    
</system.web>
<system.net>
<defaultProxy enabled="true" />
</system.net>
<!-- IIS 7.0 --> <system.webServer>
<modules>
  <remove name="ServiceModel" />
</modules>
<handlers>
  <clear />
  <add name="SvnBridgeHandler" path="*" verb="*" responseBufferLimit="0" type="SvnBridgeServer.SvnBridgeHttpHandler" resourceType="Unspecified" preCondition="integratedMode" />
</handlers>
<security>
  <requestFiltering allowDoubleEscaping="true">
    <requestLimits maxAllowedContentLength="262144000" /> <!-- 250 MB -->
    <fileExtensions>
      <clear />
    </fileExtensions>
    <hiddenSegments>
      <clear />
    </hiddenSegments>
  </requestFiltering>
</security>
</system.webServer>
</configuration>
Mar 16, 2016 at 5:00 PM
lukaszp1985 wrote:
I am building your project in VS Express 2015 for Desktop. After open solution from codeplex source I get error:
``` Unsupported
This version of Visual Studio is unable to open the following projects. The project types may not be installed or this version of Visual Studio may not support them.
For more information on enabling these project types or otherwise migrating your assets, please see the details in the "Migration Report" displayed after clicking OK.
 - SvnBridgeViaIis, "D:\temp\SvnBridge-preview\SvnBridge-preview\SvnBridgeViaIis\SvnBridgeViaIis.csproj"
Hmm, ok, the SvnBridgeViaIis project (among others) had been deleted in upstream sources, possibly for this reason of MSVS2015 barfing, but I restored it since there was no reason to be seen whatsoever (and that behaviour as seen in MSVS2015 here also certainly is not a reason for roughshod deletion of that project).
But then I can build project SvnBridge and run from bin file SvnBridge.exe. But I can't use SvnBridge desktop mode, because TFS 2013 is available on network location (via http://10.0.1.100:8080/tfs), so I need SvnBridge Server (my ip: 10.0.1.200).
Err....... why?
Of course TFS servers usually are available on network, but that certainly does not mean that TFS <-> SVN interfacing as offered by SvnBridge cannot be provided by the desktop mode SvnBridge variant (and, optionally, then even provided publically, by semi-dangerously not restricting to localhost-only network interface bind).
After configure IIS I try to use TortoiseSVN repo-browser with URL http://10.0.1.100:8083/DefaultCollection/Project and get error: Error running context. An error occurred during authentication.
If I try open website locally (localhost:8083) I get 401.2 error (lang PL):
Translated from Polish: You do not have permission to view this page. The reason is bad authentication headers.
At least in case of desktop SvnBridge mode towards TFS2013, the URL style usually to be used is:
http://<SVNBRIDGE_HOST>:<SVNBRIDGE_PORT>/<TFS_HOST>:<TFS_PORT>/tfs/DefaultCollection/$/MyTeamProj

TFS web access interface should also be successfully reachable (testing!!!) via browser (e.g. Konqueror on Linux) via URL style
http://<SVNBRIDGE_HOST>:<SVNBRIDGE_PORT>/<TFS_HOST>:<TFS_PORT>/tfs/DefaultCollection/MyTeamProj
(without "$/" TFS root component)
To access to repo by VS I have to provide login and password for my domain user.
...which needs to be provided via SVN-side auth, which is directly transparently forwarded (yet potentially insufficiently securely hashed or even plaintext).
Also I have some questions:
Do I have to run Setup.reg from SvnBridgeServer directory?
Frankly I don't know, or IOW "I believe it was not required for my use cases" (but perhaps that ASP.NET-related registry tweak in fact had to be applied initially).
What is DomainIncludesProjectName parameter from Web.config?
SvnBridgeServer/SvnBridgeHttpHandler.cs:
        if (Configuration.UseCodePlexServers)
        {
            pathParser = new PathParserProjectInDomainCodePlex();
        }
        else if (Configuration.DomainIncludesProjectName)
        {
            pathParser = new PathParserProjectInDomain(Configuration.TfsUrl, Container.Resolve<TFSSourceControlService>());
        }
        else if (!string.IsNullOrEmpty(Configuration.TfsUrl))
        {
            pathParser = new PathParserSingleServerWithProjectInPath(Configuration.TfsUrl);
        }
        else
                    {
            pathParser = new PathParserServerAndProjectInPath(Container.Resolve<TfsUrlValidator>());
                    }
        dispatcher = new HttpContextDispatcher(pathParser, Container.Resolve<ActionTrackingViaPerfCounter>());

The PathParser is instantiated per each SvnBridge use case variant (either desktop SvnBridge, or IIS deployment, or ...),
and AFAIK it adjusts TfsTeamProject location details in regards to different TFS setup variants.
Which address is correctly to enter in TortoiseSVN repo-browser: http://10.0.1.100:8083/tfs/DefaultCollection/Project or without tfs - http://10.0.1.100:8083/DefaultCollection/Project?
I'm not quite sure here, but I believe that depends on the URL location as custom-configured for a TFS setup, which also is differently-defaulted for different TFS versions (newer ones prefer to have "tfs" added I think).
Server with TFS info:
Windows 7 Pro
IIS 7.5.7600.16385
IP: 10.0.1.100
IIS app port: 8083

Client:
TortoiseSVN 1.9.3, Build 27038 - 64 Bit , 2015/12/13 11:53:16
IP: 10.0.1.200
That client ought to be sufficiently compatible, I think, given that I did some compat fixes for newer versions.
Web.config:
I cannot say much about IIS-variant-related Web.config setup details currently, since I don't have ready access to these aspects here.


Thank you for the very detailed and useful questions!
(albeit answered in a somewhat less informative manner, I have to admit)

HTH!
Mar 18, 2016 at 1:05 PM
Edited Mar 18, 2016 at 1:06 PM
I have tried desktop mode and it works!
I have successfully do SVN Checkout via http://localhost:9090/10.0.1.100:8080/tfs/DefaultCollection/$/Project
So for now I don't need SvnBridge Server.

There is one problem (bug?). After SVN Checkout I do some changes in one file and execute Check In via Visual Studio. And then I click SVN Update on that file and nothing changes in file. If I click SVN Update on whole directory everything is ok- file is updated correctly.

Many thanks for new version!
Mar 18, 2016 at 2:30 PM
Very nice to hear that!

And especially thanks for reporting this issue!
Use cases of various people/environments are very different, thus bugs in various areas are difficult to detect, thus it is very important to have proper feedback.
I have recorded this issue, and I will verify what needs to be done with my already quite improved local version (perhaps that already works, else....).

I intend to update public version relatively soonish, but only once it's sufficiently stabilized.



On more general terms, of course one might prefer to use certain other TFS interfacing methods, which should be (more or less complete list?):
a) "git tf"
b) "git tfs"
c) git team projects (initial configuration choice) in newer TFS versions
d) SvnBridge
e) Team Explorer Everywhere https://www.visualstudio.com/de-de/products/team-explorer-everywhere-vs.aspx


I don't know much about "git tf" or "git tfs" (only that for one particular purpose here one of these performed plain miserably), and neither about Team Explorer Everywhere (this might be an alternative if it's strictly about an SVN-only-style workflow, but if one is used to a much more elegant git workflow, then that one is plain incompatible, plus it also needs local-host toolkit stuff rather than simply svn-protocol-consuming clients).

So, SvnBridge IMHO has e.g. the following major benefits:
  • not restricted to git-only protocol (since it's SVN, it may be used with all SCM tools which are able to interface to SVN, be it git-svn or Mercurial or SVN or whatever)
  • transparently exposes plain SVN protocol on network side - no client-side toolkit shenanigans needed (such as Java or - heaven forbid - Mono...)
When using git, one may simply do very efficient and boring background git-svn fetching of new revisions, e.g. via a cron job which does that every 15 minutes or so,
then deciding to checkout new revisions when/as needed.
Mar 18, 2016 at 3:25 PM
To list of TFS interfacing methods you can add of course TFS (current version is 2015) with TFS Power Tools which add shell integration with Windows Explorer. This install Visual Studio Shell and lot of other apps.
For me SvnBridge is definitely best solution to work with SVN and TFS repositories.
Oct 20, 2016 at 12:18 PM
Hello! Can you create small compilation and installation tutorial ?
Because i still getting compilation error in SvnBridgeViaIis with VS2013 and TFS2013 working with errors through TortoiseSVN.

After successful checkout i tried commit and got "Error running context: The server sent an improper HTTP response"
Oct 20, 2016 at 2:09 PM
Details of this error:
System.IO.IOException: Unable to write data to the transport connection: Программа на вашем хост-компьютере разорвала установленное подключение. ---> System.Net.Sockets.SocketException: Программа на вашем хост-компьютере разорвала установленное подключение
   at System.Net.Sockets.Socket.Send(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
   at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   at System.IO.MemoryStream.WriteTo(Stream stream)
   at SvnBridge.Net.ListenerResponseStream.ForwardStreamBuffer() in d:\Source\SvnBridge-preview\SvnBridge-preview\SvnBridgeLibrary\Net\ListenerResponseStream.cs:line 193