fbpx
Welcome, Guest
Username: Password: Remember me
  • Page:
  • 1
  • 2

TOPIC:

X# slow - maybe the array handling ? 12 Nov 2020 16:22 #16621

  • Horst's Avatar

  • Horst

  • Topic Author


  • Posts: 172
  • Hello
    I uploaded a part of prg. In this part i am reading a DB sequential and building a array. The DB has only 1000 records.
    VO needs 0.42 seconds and X# incredible 5.08 seconds
    How can i make it faster? I think its the aadd() function.

    Attachment not found



    Horst
    Attachments:

    Please Log in or Create an account to join the conversation.

    X# slow - maybe the array handling ? 12 Nov 2020 16:32 #16622

  • Chris's Avatar

  • Chris


  • Posts: 2258
  • Hi Horst,

    There are a lot of things going on in this code, hard to tell what's causing the slowness, without being able to run it. Please provide a compileable and runnable sample, so we can investigate the issue.
    XSharp Development Team
    chris(at)xsharp.eu

    Please Log in or Create an account to join the conversation.

    X# slow - maybe the array handling ? 12 Nov 2020 17:14 #16623

  • Horst's Avatar

  • Horst

  • Topic Author


  • Posts: 172
  • Hi Chris
    i removed the Aadd () and the app-part has still 4.8 seconds
    I will try to make a example app with the dbf.

    Horst

    Please Log in or Create an account to join the conversation.

    X# slow - maybe the array handling ? 12 Nov 2020 17:59 #16624

  • wriedmann's Avatar

  • wriedmann


  • Posts: 2451
  • Hi Horst,
    AFAIK Robert is working on a faster read of DBF files.
    VO is relative aggressive in caching, and apparently VFP even more.
    The first target in the development of the X# RDD was compatibility over speed, but since also speed is an issue, the development team has worked on this also.
    As far as I had understand, the X# RDD will be a bit slower than the one from VO because it is written in (slower) managed code, and that is the price for the compatibility with x64/AnyCPU.
    But I have to admit that I also have speed issues with the VO RDD specially with more users on networks, and in these cases I'm using the Advantage Database Server.
    Wolfgang
    Wolfgang Riedmann
    Meran, South Tyrol, Italy

    www.riedmann.it - docs.xsharp.it

    Please Log in or Create an account to join the conversation.

    X# slow - maybe the array handling ? 13 Nov 2020 02:43 #16626

  • Jamal's Avatar

  • Jamal


  • Posts: 231
  • The DBFCDX Rdd is extremely slow as of XSharp Cahors 2.6a and prior versions . See my post from September 15:

    www.xsharp.info/forum/public-product/214...s-vo-huge-difference

    The tests included there will a good test case to, hopefully, see if the next version will speed it up.

    Please Log in or Create an account to join the conversation.

    Last edit: by Jamal.

    X# slow - maybe the array handling ? 13 Nov 2020 09:55 #16629

  • robert's Avatar

  • robert


  • Posts: 1980
  • Jamal,
    For the next build we have improved the RDD speed in Exclusive mode A LOT.

    In one of the builds after that we will try to improve the speed of shared mode as well, but that may cause compatibility problems.
    FoxPro seems to simply assume nothing has changed and shows "old data" even when another user has changed the data in the mean time.
    It waits until you try to update the data before checking for changes made by other users.
    And after a timeout (SET REFRESH TO) it refreshes the local caches of the data.

    One of the things that has a negative impact on the speed, even in exclusive mode, is that the data inside DBF files is always Ansi and we have to convert it to Unicode.

    A positive side effect of the changes that we did is that we can now also access DBF files larger than 2 Gb.
    Unfortunately that may cause problems with shared access since the locking scheme then overlaps with the record contents.
    We will add a (configurable) option to switch to Harbour locking scheme for large files when your DBF exceeds the 2 Gb.

    Robert
    XSharp Development Team
    The Netherlands

    Please Log in or Create an account to join the conversation.

    X# slow - maybe the array handling ? 13 Nov 2020 10:04 #16630

  • wriedmann's Avatar

  • wriedmann


  • Posts: 2451
  • Hi Robert,
    that is great to hear, thank you very much!
    Regarding the speed in the shared mode: I have noted that also ADS is slow committing changes, so sometimes even in the same program instance you will see old values. Therefore I had to add small delays in some places of my program.
    Theoretically a look to the header of the DBF file should be enough so see if a program has changed the data, but it will always be a bit of risk - and only a server engine like a SQL database will be able to solve that (what makes me return to the idea of XDS - the X# Database Server <g>).
    Wolfgang
    Wolfgang Riedmann
    Meran, South Tyrol, Italy

    www.riedmann.it - docs.xsharp.it

    Please Log in or Create an account to join the conversation.

    X# slow - maybe the array handling ? 13 Nov 2020 11:01 #16631

  • robert's Avatar

  • robert


  • Posts: 1980
  • Wolfgang,
    I am afraid that a look in the DBF header is not enough:
    - The DBF header only changes when a new record is appended.
    - The DBF header does not change when a record was updated, unless the last change was on a different day. In that case the LastUpdate field is changed.
    - The CDX and NTX header however always get updated when an index key was changed.

    However...
    There is a 4 byte field in the DBF header (DBASE_LAN, only used by Dbase AFAIK) that we could use for a version counter:
    github.com/X-Sharp/XSharpPublic/blob/fea...bf/DbfHeader.prg#L31
    We may be able to increment this field for every update to a DBF, so other stations can see that the file was updated.

    That means that this page from the DBF should never be cached but all the other pages could be cached, as long as the version number in this location is unchanged.

    I'll see what I can do.

    Robert
    XSharp Development Team
    The Netherlands

    Please Log in or Create an account to join the conversation.

    X# slow - maybe the array handling ? 13 Nov 2020 11:20 #16632

  • wriedmann's Avatar

  • wriedmann


  • Posts: 2451
  • Hi Robert,
    that would be great!
    Wolfgang
    Wolfgang Riedmann
    Meran, South Tyrol, Italy

    www.riedmann.it - docs.xsharp.it

    Please Log in or Create an account to join the conversation.

    X# slow - maybe the array handling ? 13 Nov 2020 11:49 #16633

  • Horst's Avatar

  • Horst

  • Topic Author


  • Posts: 172
  • Hello
    So i will have passion for the next update. in the meantime i saw some things i can write different to make it faster.
    i am making now a standalone exe , so i can test the speed later with the same code and the new x# update.

    Horst

    BTW the X# team is realy making a good job, and they have alsp passion with one like me ;-) Thanks

    Please Log in or Create an account to join the conversation.

    X# slow - maybe the array handling ? 16 Nov 2020 13:00 #16654

  • ic2's Avatar

  • ic2


  • Posts: 819
  • I am happy to hear that too. Currently DBF handling and editor speeds are IMO the most urgent issues while so much else is working very well so it's good it has your full attention.

    But these 2 issues are for most users the most important.

    Dick

    Please Log in or Create an account to join the conversation.

    X# slow - maybe the array handling ? 16 Nov 2020 14:09 #16657

  • robert's Avatar

  • robert


  • Posts: 1980
  • Dick,

    I do not agree that DBF speed is for most users the most important.
    Many, many users have switched away from DBF to client server solutions using either Advantage or one of the SQL solutions available.

    In a networked environment DBF based solutions are no longer recommended anymore because of all the SMB caching issues which force you to tweak registry settings.

    This does not mean that we will not look at this though.

    Robert
    XSharp Development Team
    The Netherlands

    Please Log in or Create an account to join the conversation.

    X# slow - maybe the array handling ? 16 Nov 2020 14:35 #16658

  • ic2's Avatar

  • ic2


  • Posts: 819
  • Hello Robert,

    We use ADS too, as do almost all our clients. But as Wolfgang also indicates, ADS with X# also has some speed issues. So in a broader sense:

    DBF/ADS & using X# in VS are (IMO) the most important issues. If that works flawlessly, speed & functionality, everything else X# can do above VO is (for us) "nice to have", at most.

    And I do realize very well how hard you are working on it. Therefore I keep renewing our Fox programs despite we still use very little of X# and most of the (admittedly elaborate) functionality which has been and will be added.

    Dick

    Please Log in or Create an account to join the conversation.

    X# slow - maybe the array handling ? 16 Nov 2020 14:39 #16659

  • wriedmann's Avatar

  • wriedmann


  • Posts: 2451
  • Hi Dick,
    please let me confirm that the speed issues that ADS sometimes has have nothing to do with X#, but they are also occurring when using VO.
    Nevertheless we need also the DBF access in networks because we have many customers we cannot switch over to ADS (even if I would like to do, but currently it seems X# will have a longer life than ADS).
    Wolfgang
    Wolfgang Riedmann
    Meran, South Tyrol, Italy

    www.riedmann.it - docs.xsharp.it

    Please Log in or Create an account to join the conversation.

    X# slow - maybe the array handling ? 16 Nov 2020 16:55 #16661

  • ic2's Avatar

  • ic2


  • Posts: 819
  • Hello Wofgang,

    We also had issues (cdx files got corrupted) using our X# app in a CGI environment (obviously without ADS). These may be solved by now but as it needed to get solved (it was in a production environment) we moved back to the Vulcan DLL's and now it works without problems. That's what I mean with that in general I consider DBFCDX/ADS + the VS integration so important.

    Note that you can also use the ADS local server for users who do not (want to buy) the ADS Remote Server. Officially no more than 5 simultaneous users are supported but from experience I know that it works for more.

    Dick

    Please Log in or Create an account to join the conversation.

    Last edit: by ic2.

    X# slow - maybe the array handling ? 16 Nov 2020 16:58 #16662

  • wriedmann's Avatar

  • wriedmann


  • Posts: 2451
  • Hi Dick,
    the ADS Local Server also does not works on a remote desktop session.
    I have also a license for the Apollo Driver, but have yet to test it on a database shared with VO.
    Wolfgang
    Wolfgang Riedmann
    Meran, South Tyrol, Italy

    www.riedmann.it - docs.xsharp.it

    Please Log in or Create an account to join the conversation.

    X# slow - maybe the array handling ? 17 Nov 2020 00:14 #16665

  • ic2's Avatar

  • ic2


  • Posts: 819
  • Hello Wolfgang,

    wriedmann wrote: ,
    the ADS Local Server also does not works on a remote desktop session.


    It surely does. In the ADS.ini you have to add:

    MTIER_LOCAL_CONNECTIONS=1

    It turned out to work when we were unable to find out why ADS Remote server wasn't recognized on an unusual server configuration which was setup by a dealer after disk space expansion extremely slowed down program execution.

    This is supposed to be a test only statement but without decent support of SAP we had to try that and it did actually allow that client of ours continue working on their RD, with local server.

    Dick

    Please Log in or Create an account to join the conversation.

    X# slow - maybe the array handling ? 18 Nov 2020 12:52 #16689

  • mainhatten's Avatar

  • mainhatten


  • Posts: 151
  • Robert,

    robert wrote: FoxPro seems to simply assume nothing has changed and shows "old data" even when another user has changed the data in the mean time.
    It waits until you try to update the data before checking for changes made by other users.
    And after a timeout (SET REFRESH TO) it refreshes the local caches of the data.

    In vfp the idea of local caching of "remote" (might just be ISAM-accessed dbf on file server) persistance-needing data was born when 10MB nets and ISDN online speeds were common here ;-)
    Combined with Rushmore optimizing most data accesses a lot best practices oriented on buffering, (usually optimistic) locking strategy, conflict resolution options and chances to look at needed time with curval(). The mindset on shared table handling was that of a SQL backend data store.

    One of the things that has a negative impact on the speed, even in exclusive mode, is that the data inside DBF files is always Ansi and we have to convert it to Unicode.

    Compared to ANSI runtimes like vfp or VO this is more effort. OTOH doubling char field sizes to store 2Byte "chars" would make transfer from disk to memory slower.

    A positive side effect of the changes that we did is that we can now also access DBF files larger than 2 Gb.
    Unfortunately that may cause problems with shared access since the locking scheme then overlaps with the record contents.
    We will add a (configurable) option to switch to Harbour locking scheme for large files when your DBF exceeds the 2 Gb.


    ADS also has the option to use tables beyond 2GB and implements a different locking schema on those tables. Have you compared Harbour and ADS locking method ? If they are different, ADS-opened large tables should not be written into from xSharp...

    regards
    thomas

    Please Log in or Create an account to join the conversation.

    Last edit: by mainhatten.

    X# slow - maybe the array handling ? 18 Nov 2020 13:27 #16690

  • mainhatten's Avatar

  • mainhatten


  • Posts: 151
  • robert wrote: However...
    There is a 4 byte field in the DBF header (DBASE_LAN, only used by Dbase AFAIK) that we could use for a version counter:
    github.com/X-Sharp/XSharpPublic/blob/fea...bf/DbfHeader.prg#L31
    We may be able to increment this field for every update to a DBF, so other stations can see that the file was updated.
    That means that this page from the DBF should never be cached but all the other pages could be cached, as long as the version number in this location is unchanged.

    If working reliably this could short-circuit CursorRefresh calls if vfp. But on heavy used tables a 4Byte integer might overflow and any schema to reset might have trouble with cursors explicitly set offline (unless they get special treatment in "Use Online")
    Also concurrent use with VO or vfp, which will not set that counter, must be prohibited - new dbf type set in header?

    regards
    thomas

    Please Log in or Create an account to join the conversation.

    Last edit: by mainhatten.

    X# slow - maybe the array handling ? 18 Nov 2020 13:42 #16691

  • mainhatten's Avatar

  • mainhatten


  • Posts: 151
  • Hi Wolfgang,

    wriedmann wrote: please let me confirm that the speed issues that ADS sometimes has have nothing to do with X#, but they are also occurring when using VO.

    No code seen so no idea what might happen - but I think the MS ideas on disconnected result sets, buffering, locking strategies and where clause construction for update were great. You can find the vfp help file for download on vfpX - if you have not done already, read the parts on buffering, locking and updating views (which is identical to the later introduced Cursoradapters). You can also check on update strategies in Dotnet ADO.Net Datasets and Datatables - same options for where, only not as a single ENUM with 4 values but 2 properties IIRC. Godd stuff to ponder if you have not checked already.

    Nevertheless we need also the DBF access in networks because we have many customers we cannot switch over to ADS (even if I would like to do, but currently it seems X# will have a longer life than ADS).

    As long as it works, calling it dead is premature - vfp got no maintainance and is still "alive" ;-)

    regards
    thomas

    Please Log in or Create an account to join the conversation.

    • Page:
    • 1
    • 2