Existing Base Import Time, Archive Time / Initial Archiving Method

PDM Generality
1

Hello

 

I'm in  the process of trying to archive my 80GB of existing solidworks files.

I work directly remotely on the archive server (virtualized server rather very high-end).

Here is an estimate of the times I get:

- copy of SW files from the network to the local view of the vault: about 10 hours for 80GB or a 2MB/s (this is not to be compared to network to disk copy time because EPDM does more operations than the simple copy: creation of a file in the vault in addition to the one in the local view, creating entries in the SQL database ..)

- first archiving of files: about 40 archived parts files per minute (there is a workflow to automatically fill in the file's index in EPDM so some files end up at version 5 at the end of the archiving)

Do you find these copy times and archive times normal?

 

The first archiving is also very tedious, SW and resellers recommend importing the SLDPRTs first, then the SLDASMs and then the SLDDRWs. For this I either used a dispatch (see my question on the subject) but it stops at each error/warning (file not found, duplicate ....) so it's not practical. I also tried to use the search to filter and archive a certain type of files but having a database that is too big it is not possible to do this on the whole database and doing this directory by directory (I have 1000 plans per directory) is very long and painful. If I am forced to use the latter method, the archiving necessary before I can use EPDM will last several days, during which I will put my colleagues on technical unemployment...

Is there a simple and quick method to do this first archiving (if I can also get the list of errors/warnings in a file at the same time it would be the dream).

Thank you for your feedback

 

2

During our time at EPDM, we (@Benoit.LF and I) had a lot of data to process like you.

We had quite a bit of "cleanup" of the data beforehand via the INTEGRATION utility to avoid any errors during EPDM archiving.

We first imported the SLDPRT+SLDDRW and then the SLDASM.

Imports into the EPDM were done in the evening and at night and on weekends (in packs of 1000 references), once imported into the EPDM these non-EPDM references were blocked in read-only mode (so it was impossible for our colleagues to make the slightest modification on these parts).

From home via TeamViewer, we took control of all or almost all of our colleagues' workstations to save as much time as possible.

Once imported and archived, via an import WF (to match, in particular, the index property with the EPDM index) the parts and drawings were WAITING FOR VALIDATION, and over time after a quick visual check the files went to the VALID state.

In terms of time I don't remember, but from memory it was quite long.

1 Like
3

@Flegendre

Thank you for your feedback.

I was wondering how to avoid blocking the BE during the EPDM switchover. Your solution of making the files imported into EPDM read-only on the old server allows you to respond to this problem although it implies a strongly degraded operation of the BE as long as the complete failover has not been made.

Currently I work directly on the archive server. According to your message, it seems that you went through several local stations to do the initial archiving of the files. Think your method was faster than working on the server alone? Maybe the fact of working with 2 people in parallel (waiting 1 to 2 minutes for the right click to respond on a selection of 500 to 1000 files is a long time)?

4

Indeed, we went through the workstations and not directly through the server.

Theoretically directly from the server you should be faster. We hadn't done the test between the 2 methods so on this point I can't answer you.

Ideally, it's to spend everything in one or two weekends to get as little trouble as possible...

5

Back to the import time of the database:

On a database of 42 GB of data (46096 Files, 4503 Folders), it took me 2 and a half days to import the files (copy/paste in the vault: about 3 to 4 hours), then do the first check in in the directory dedicated to the import (about 2 days with check in  of the sldprt then the sldasm then the slddrw, then all the other files lying around in the directories (office, pdf, igs ...)).

To do the check-in make use of the EPDM search tool and not the search in the explorer (much slower and crashes more). My times are quite long because I import almost all my properties related to the configurations and because I have a rather complicated initial workflow (it automatically retrieves the revision of the files according to the value of the " revision " property)

With the search tool do batch check-ins of 2000 to 5000 files (chances are that the check-in plants from time to time and since the reference search time is quite long, it's better not to take too many files at once). There is the possibility of doing several searches and several check-ins in parallel on the same machine. On the other hand, the searches must be independent to avoid crashing a batch of check-ins because EPDM tries to do a check-in on a file that is already in the safe: on parts and MEP files it works well by doing searches such as 'S*.sldprt', on assemblies it is riskier because a sub-assembly can have a name that has nothing to do with the initial assembly

Do all the import and check-in manipulations on the archive server directly.

Being in a virtualized environment I temporarily boosted the EPDM servers: switching to 4 dedicated cores for the SQL server (not necessarily very useful in the end because I had a CPU utilization rate of about 20%, 2 cores would surely have been enough).

Once the import and the initial check in is done in the dedicated directory, you have to put the directories back in their place. This should be done ONLY on a machine with an empty local cache. If you do this on the server containing all the files on its local view, it's horribly long (you have to move GB of data and EPDM may update the data during the transfer so that the links are OK when opening the files). If the local view is empty, nothing happens on the workstation and only the SQL server works to change the file location information (only one transaction per file moves).

 NB: my American reseller Trimech (which bought Moderntools) now has a tool to analyze SW directories before importing. This tool scans SW directories and outputs a database containing all the links between files, the properties contained in the files.... (it was still several hundred MB). From there, it is possible to correct all SW files containing invalid references, incorrectly filled properties (entered on the file instead of on the configurations for example...) via a number of other tools. I haven't been able to see these tools in operation so I don't know how easy they are to use and how ergonomic they are. My dealer didn't want to show me anything if we didn't go in this direction, since everything had already been done for the 'traditional' import I didn't want to embark on this adventure. On the other hand, this kind of tool can be useful to a company that absolutely wants 100% of the data entered in EPDM to be compliant (this is not our case since we have very little reuse : we have therefore entered into EPDM data that is far from clean but that we will only clean according to a real need).

I don't know if Axemble has similar tools.

 

I hope this topic will be useful to future brave EPDM admins when they first import.