fbpx
Welcome, Guest
Username: Password: Remember me
  • Page:
  • 1

TOPIC:

DBF Append Speed in X# vs VO - Huge Difference 15 Sep 2020 07:11 #15878

  • Jamal's Avatar

  • Jamal

  • Topic Author


  • Posts: 224
  • Hi Dev Team!

    I created a small app that collects data from a webservice and adds 100 records every 3 minutes. The file reached millions of records and the .NET app started to slow down (I thought it was freezing since there was no indication what is going behind the scene), then after extensive CPU profiling and debugging, I isolated the issue to a single function in the DBFCDX RDD.

    The X# DBF DBAppend function (or dbSever Append method - also tested) is very, very slow compared the blazing speed in VO. Also, when I used the Advantage AXDBFCDX, it was very fast!

    To test I created a dummy dbf with one field ( Char(20) ) and added 1 million records.
    On my system, X# took 63.27 seconds while VO only took 0.47 milliseconds. Simpy call the DBAppend function in a loop.

    Using X# public build: 2.5b
    VO 2.8 - 2838
    Windows 10 Pro, SSD drive.

    Here is the code:
    FUNCTION Start() AS VOID
    	LOCAL cDBF AS STRING 
    	LOCAL aFields AS ARRAY 
    	LOCAL i AS DWORD	
            LOCAL nStart, nElapsed AS FLOAT
        
    	RddSetDefault ( "DBFCDX" ) 
    	
    	cDbf := "C:\test\mytest"    // change the dbf to "C:\test\mytestVO"  to test in VO 2.8.
    	FErase ( cDbf + ".cdx" )
    	
    	aFields := { { "LAST" , "C" , 20 , 0 } } 
    	
    	
    	DbCreate( cDBF , AFields)
    	DbUseArea( ,,cDBF )		
        
            nStart := Seconds()
        
           ? "start : " + ntrim(nStart)
        
    	FOR i := 1 UPTO 1000000
    		DbAppend() 		
    	NEXT
    	
           nElapsed := Seconds() - nStart
    
           ? "Elapsed: " + NTrim(nElapsed) + " seconds"
       
    	DbCloseArea()
    		
        wait
    RETURN

    Please Log in or Create an account to join the conversation.

    Last edit: by Jamal.

    DBF Append Speed in X# vs VO - Huge Difference 15 Sep 2020 08:33 #15879

  • robert's Avatar

  • robert


  • Posts: 1862
  • Jamal,
    We are aware of this and we are working on it:
    There is a glitch in the page splitting algorithm .
    The resulting index is correct but contains many pages with only one key per page.
    This seems to be related to the large number of duplicate key values (in your case 1000000 blank keys).

    Robert
    XSharp Development Team
    The Netherlands

    Please Log in or Create an account to join the conversation.

    DBF Append Speed in X# vs VO - Huge Difference 15 Sep 2020 08:57 #15880

  • Chris's Avatar

  • Chris


  • Posts: 2126
  • Robert,

    I think this is a different issue, because there is no index file involved. Something seems to make appending records much slower in the .Net version.
    XSharp Development Team
    chris(at)xsharp.eu

    Please Log in or Create an account to join the conversation.

    DBF Append Speed in X# vs VO - Huge Difference 15 Sep 2020 19:16 #15885

  • Jamal's Avatar

  • Jamal

  • Topic Author


  • Posts: 224
  • Robert,

    In the sample I provided I am just appending blank records; they are not duplicates. If I create an index and use FIELDPUT( ) in the LAST field, the process in X# is extremely slow.

    Please see the modified sample code below.It took a whopping 57 minutes in X#, while VO only took 3.49 seconds!


    So there are two issues, Append and the page splitting algorithm you mentioned:
    FUNCTION Start() AS VOID
    	LOCAL cDBF AS STRING 
    	LOCAL aFields AS ARRAY 
    	LOCAL i AS DWORD
    	
            LOCAL nStart, nElapsed AS FLOAT
        
    	RddSetDefault ( "DBFCDX" ) 
    	
    	cDbf := "C:\test\mytest" 
    	FErase ( cDbf + ".cdx" )
    	
    	aFields := { { "LAST" , "C" , 20 , 0 } } 
    	
    	
    	DbCreate( cDBF , AFields)
    	DbUseArea( ,,cDBF )	
        
            DBCreateOrder("last", "c:\test\mytest", "last")
        
           nStart := Seconds()
        
           ? "start : " + ntrim(nStart)
        
    	FOR i := 1 UPTO 1000000
    		DbAppend() 		
                    fieldput(1, asstring(i))
    	NEXT
    	
           nElapsed := Seconds() - nStart
    
           ? "Elapsed: " + NTrim(nElapsed) + " seconds"
    
        
    	DbCloseArea()
    		
        wait
    RETURN

    Please Log in or Create an account to join the conversation.

    Last edit: by Jamal.

    DBF Append Speed in X# vs VO - Huge Difference 16 Sep 2020 10:10 #15888

  • robert's Avatar

  • robert


  • Posts: 1862
  • Jamal,
    We'll look at this asap. Not for the 2.6 build that is coming in a few days, but in the build after that.
    I know that VO used very aggressive caching for files opened exclusively, like in your example.
    We'll try to build something similar in X#.

    Robert
    XSharp Development Team
    The Netherlands

    Please Log in or Create an account to join the conversation.

    DBF Append Speed in X# vs VO - Huge Difference 17 Sep 2020 01:20 #15889

  • Jamal's Avatar

  • Jamal

  • Topic Author


  • Posts: 224
  • Robert,

    Will test again when you issue the fix and I'll report back.

    Thanks!

    Please Log in or Create an account to join the conversation.

    DBF Append Speed in X# vs VO - Huge Difference 17 Sep 2020 13:35 #15899

  • ic2's Avatar

  • ic2


  • Posts: 755
  • This seems to be related to the large number of duplicate key values (in your case 1000000 blank keys).

    We were actually the X# users having that issue that many duplicate values corrupted the CDX after mostly 1 or 2 days of use in a CGI webprogram we fully converted to X#. So I am surprised that this issue still exists as I thought it was solved.

    However, as we had another CDX corruption issue and as much as we would have preferfed using the native X# ADS/DBF drivers, we couldn't let the client keep functioning as guinea pig :P we eventually reverted the DBFCDX system back to the Vulcan method. Since then the X# webprogram works fine.

    So I was wondering Jamal: do you have the same speed issue if you use the Vulcan DLL/RDD's (only for the DBFCDX functionality)?

    Dick

    Please Log in or Create an account to join the conversation.

    Last edit: by ic2.

    DBF Append Speed in X# vs VO - Huge Difference 17 Sep 2020 18:17 #15902

  • Jamal's Avatar

  • Jamal

  • Topic Author


  • Posts: 224
  • Dick,

    I don't have Vulcan anymore and so I did not test it.

    Jamal

    Please Log in or Create an account to join the conversation.

    DBF Append Speed in X# vs VO - Huge Difference 22 Sep 2020 10:07 #15941

  • KeesIC2's Avatar

  • KeesIC2


  • Posts: 13
  • I think all the changes and improvements in 2.6 are less important than a reliable DBF/CDX system. I can't use X# because indexes get corrupted even when using only basic functionality. Yet more and more new features are added while this is just not working properly.

    Please Log in or Create an account to join the conversation.

    Last edit: by KeesIC2.

    DBF Append Speed in X# vs VO - Huge Difference 22 Sep 2020 10:40 #15943

  • robert's Avatar

  • robert


  • Posts: 1862
  • Kees,
    We have already made some changes in this area, so most likely the problem is fixed.
    Did you test the new version?

    Robert
    XSharp Development Team
    The Netherlands

    Please Log in or Create an account to join the conversation.

    • Page:
    • 1