Issue link: https://geospatial.trimble.com/en/resources/i/1415711
Trimble Software: Working With Large Datasets 6 Data Transfer ● Raw data formats are frequently more compact than project files after importing into the office software. o Consider the data size differences when determining how you want to move and share the data between computers. Use raw data when possible to save time. ● Multiple users could be processing different parts of the same project. o Moving data between the processing machines needs to be efficient. o If everyone is local to one office it is frequently faster to pass around high-speed portable drives rather than using a network, unless special high-speed network connections to local servers (e.g., optical fiber or Terabit Ethernet connections) are available to each of the local machines. ● Portable hard drive speed is very important. o Selecting a portable hard drive based on its speed rather than price can help to prevent costly project delays. Be sure to purchase a drive that is fast enough to avoid long wait times while transferring data. Speeds in the hundreds of Mbps and above are suggested. You may need to find 3 rd party reviews to get this information rather than manufacturer's specifications. o USB protocols (USB 2.0, USB 3.x …) specify the maximum amount of data that can flow through the interface. The protocol used does not necessarily indicate the speed of the drive itself. o Solid state drives are recommended because they typically have the transfer speed required for use with large data sets. o A USB-C connected NVMe drive (within an external enclosure) is also a recommended option that can provide very fast data transfer. Processing & Deliverable Creation ● It is important to keep track of your data flow while working on large data sets. o Frequently multiple copies of a project are generated while trying different methods of processing. It is easy to lose track of which changes were made to the copies of the project. o Naming/versioning of the copies is important to minimize confusion. Where possible, use naming conventions that describe the differences between the different copies. o Multiple copies of a project can consume large amounts of drive space. This is especially true when producing many different scan data registrations within a given mobile mapping project. ● Dividing up a project during processing will sometimes improve efficiency. o This applies most to large scanning and photogrammetry projects. o Focusing on processing smaller pieces of the project, one at a time, can help to reduce confusion for larger overwhelming projects. o Care must be taken to plan how the project is to be divided for processing and even more importantly, how the results will be consolidated. There are only limited types of separation of project data that can be recombined in TBC and Trimble RealWorks effectively. o Don't forget to factor into the overall equation the extra data transfer time related to a divide and conquer approach. It will take time to move data between the processing computers.