You are reading the documentation for version 3.0 of ProMod3.
You may also want to read the documentation for:
1.3
2.0
2.1
3.1
3.2
Using Binary Files In ProMod3¶A few features in ProMod3 (and potentially your next addition) require binary files to be loaded and stored. Here, we provide guidelines and describe helper tools to perform tasks related to loading and storing binary files. Generally, each binary file consists of a short header and binary data. The header ensures consistency between the storing and the loading of data, while the “binary data” is some binary representation of the data of interest. The main issue, we try to address is that in C++, the binary representation
of objects can be machine- and compiler-dependent. The standard guarantees
though that
Portable I/O should always be provided for binary files. If this is too slow for your needs, you can provide functionality for raw binary files. In that case you should still distribute only the portable file and provide a converter which loads the portable file and stores a raw binary file for further use. Storing and loading of raw binary files on the same machine with the same compiler should never be an issue. For instance, the classes File Header¶The header is written/read with functions provided in the header file
For portable I/O (see below), we only write/read fixed-width fundamental
data-types (e.g. All write functions (when saving a binary) should be mirrored by the corresponding check (or get) function in the exact same order when loading. All functions are templatized to work with any OST-like data sink or source
and overloaded to work with Portable Binary Data¶Portable files are written/read with functions and classes provided in the
header file Writing and reading is performed by the following classes:
Each serializable class must define a template <typename DS>
void Serialize(DS& ds) {
// serialize element-by-element
}
Or if this is not possible for an object of type inline void Serialize(core::PortableBinaryDataSource& ds, T& t) { }
inline void Serialize(core::PortableBinaryDataSink& ds, T t) { }
Given a sink or source object
Implementation notes:
Code Example¶Here is an example of a class which provides functionality for portable and non-portable I/O: // includes for this class
#include <boost/shared_ptr.hpp>
#include <iostream>
#include <fstream>
#include <sstream>
#include <vector>
// includes for I/O
#include <promod3/core/message.hh>
#include <promod3/core/portable_binary_serializer.hh>
#include <promod3/core/check_io.hh>
using namespace promod3;
// define some data-structure
struct SomeData {
short s;
int i;
Real r;
// portable serialization
// (cleanly element by element with fixed-width base-types)
template <typename DS>
void Serialize(DS& ds) {
core::ConvertBaseType<int16_t>(ds, s);
core::ConvertBaseType<int32_t>(ds, i);
core::ConvertBaseType<float>(ds, r);
}
};
// define pointer type
class MyClass;
typedef boost::shared_ptr<MyClass> MyClassPtr;
// define class
class MyClass {
public:
MyClass(const String& id): id_(id) { }
// raw binary save
void Save(const String& filename) {
// open file
std::ofstream out_stream(filename.c_str(), std::ios::binary);
if (!out_stream) {
std::stringstream ss;
ss << "The file '" << filename << "' cannot be opened.";
throw promod3::Error(ss.str());
}
// header for consistency checks
core::WriteMagicNumber(out_stream);
core::WriteVersionNumber(out_stream, 1);
// required base types: short, int, Real (for SomeData).
// uint (for sizes)
// required structs: SomeData
core::WriteTypeSize<uint>(out_stream);
core::WriteTypeSize<short>(out_stream);
core::WriteTypeSize<int>(out_stream);
core::WriteTypeSize<Real>(out_stream);
core::WriteTypeSize<SomeData>(out_stream);
// check values for base types
core::WriteBaseType<uint>(out_stream);
core::WriteBaseType<short>(out_stream);
core::WriteBaseType<int>(out_stream);
core::WriteBaseType<Real>(out_stream);
// write string
uint str_len = id_.length();
out_stream.write(reinterpret_cast<char*>(&str_len), sizeof(uint));
out_stream.write(id_.c_str(), str_len);
// write vector of SomeData
uint v_size = data_.size();
out_stream.write(reinterpret_cast<char*>(&v_size), sizeof(uint));
out_stream.write(reinterpret_cast<char*>(&data_[0]),
sizeof(SomeData)*v_size);
}
// raw binary load
static MyClassPtr Load(const String& filename) {
// open file
std::ifstream in_stream(filename.c_str(), std::ios::binary);
if (!in_stream) {
std::stringstream ss;
ss << "The file '" << filename << "' does not exist.";
throw promod3::Error(ss.str());
}
// header for consistency checks
core::CheckMagicNumber(in_stream);
uint32_t version = core::GetVersionNumber(in_stream);
if (version > 1) {
std::stringstream ss;
ss << "Unsupported file version '" << version
<< "' in '" << filename;
throw promod3::Error(ss.str());
}
// check for exact sizes as used in Save
core::CheckTypeSize<uint>(in_stream);
core::CheckTypeSize<short>(in_stream);
core::CheckTypeSize<int>(in_stream);
core::CheckTypeSize<Real>(in_stream);
core::CheckTypeSize<SomeData>(in_stream);
// check values for base types used in Save
core::CheckBaseType<uint>(in_stream);
core::CheckBaseType<short>(in_stream);
core::CheckBaseType<int>(in_stream);
core::CheckBaseType<Real>(in_stream);
// read string (needed for constructor)
uint str_len;
in_stream.read(reinterpret_cast<char*>(&str_len), sizeof(uint));
std::vector<char> tmp_buf(str_len);
in_stream.read(&tmp_buf[0], str_len);
// construct
MyClassPtr p(new MyClass(String(&tmp_buf[0], str_len)));
// read vector of SomeData
uint v_size;
in_stream.read(reinterpret_cast<char*>(&v_size), sizeof(uint));
p->data_.resize(v_size);
in_stream.read(reinterpret_cast<char*>(&p->data_[0]),
sizeof(SomeData)*v_size);
return p;
}
// portable binary save
void SavePortable(const String& filename) {
// open file
std::ofstream out_stream_(filename.c_str(), std::ios::binary);
if (!out_stream_) {
std::stringstream ss;
ss << "The file '" << filename << "' cannot be opened.";
throw promod3::Error(ss.str());
}
core::PortableBinaryDataSink out_stream(out_stream_);
// header for consistency checks
core::WriteMagicNumber(out_stream);
core::WriteVersionNumber(out_stream, 1);
// required base types: short, int, Real
// -> converted to int16_t, int32_t, float
core::WriteTypeSize<int16_t>(out_stream);
core::WriteTypeSize<int32_t>(out_stream);
core::WriteTypeSize<float>(out_stream);
// check values for base types
core::WriteBaseType<int16_t>(out_stream);
core::WriteBaseType<int32_t>(out_stream);
core::WriteBaseType<float>(out_stream);
// write string (provided in portable_binary_serializer.hh)
out_stream & id_;
// write vector (provided in portable_binary_serializer.hh)
// -> only ok like this if vector of custom type
// -> will call Serialize-function for each element
out_stream & data_;
}
// portable binary load
static MyClassPtr LoadPortable(const String& filename) {
// open file
std::ifstream in_stream_(filename.c_str(), std::ios::binary);
if (!in_stream_) {
std::stringstream ss;
ss << "The file '" << filename << "' does not exist.";
throw promod3::Error(ss.str());
}
core::PortableBinaryDataSource in_stream(in_stream_);
// header for consistency checks
core::CheckMagicNumber(in_stream);
uint32_t version = core::GetVersionNumber(in_stream);
if (version > 1) {
std::stringstream ss;
ss << "Unsupported file version '" << version
<< "' in '" << filename;
throw promod3::Error(ss.str());
}
// check for if required base types (see SavePortable)
// are big enough
core::CheckTypeSize<short>(in_stream, true);
core::CheckTypeSize<int>(in_stream, true);
core::CheckTypeSize<Real>(in_stream, true);
// check values for base types used in Save
core::CheckBaseType<int16_t>(in_stream);
core::CheckBaseType<int32_t>(in_stream);
core::CheckBaseType<float>(in_stream);
// read string (needed for constructor)
String s;
in_stream & s;
// construct
MyClassPtr p(new MyClass(s));
// read vector of SomeData
in_stream & p->data_;
return p;
}
private:
std::vector<SomeData> data_;
String id_;
};
int main() {
// generate raw file
MyClassPtr p(new MyClass("HELLO"));
p->Save("test.dat");
// load raw file
p = MyClass::Load("test.dat");
// generate portable file
p->SavePortable("test.dat");
// load portable file
p = MyClass::LoadPortable("test.dat");
return 0;
}
Exisiting Binary Files¶The following binary files are currently in ProMod3:
During the If the stage folder is moved after compilation (e.g. Code for the generation of the binary files and their portable versions are
in the |
ContentsSearchEnter search terms or a module, class or function name. Previous topicNext topicYou are here
|