| anhongleetcode@gmail.com: Jan 21 08:59PM -0800 Hi experts, I am new to C++, I would like to have a buffer using std::vector, like ```cpp std::vector<char> buf; buf.resize(64 * 1024 * 1024 * 1024); fread(buf.data(), buf.size(), 1, fp); func(buf.data()); // consuming it ``` I like vector, but the resize will initialize the whole vector as zero. Which is unwanted, I don't even need it, it's a waste! How could I skip this? Some people warned me the following code is dangerous... Because assessing data() outside of range [data, data + size) is undefined: ```cpp std::vector<char> buf; buf.reserve(64 * 1024 * 1024 * 1024); fread(buf.data(), buf.size(), 1, fp); func(buf.data()); // consuming it ``` then what's the proper/elegant way? I need pretty big buffer so I have to dynamically allocate it. Thanks, Anhong |
| Daniel <danielaparker@gmail.com>: Jan 21 09:27PM -0800 > I like vector, but the resize will initialize the whole vector as zero. > Which is unwanted, I don't even need it, it's a waste! > How could I skip this? Some alternatives: http://andreoffringa.org/?q=uvector Personally, I wouldn't worry about it. Daniel |
| Barry Schwarz <schwarzb@dqel.com>: Jan 21 10:30PM -0800 On Mon, 21 Jan 2019 20:59:52 -0800 (PST), anhongleetcode@gmail.com wrote: >std::vector<char> buf; >buf.resize(64 * 1024 * 1024 * 1024); >fread(buf.data(), buf.size(), 1, fp); Your call to fread specifies a single object 64GB in size. How will you know exactly how much data was actually read in? Just out of curiosity, how long will it take your system to read a 64GB block? >How could I skip this? >Some people warned me the following code is dangerous... >Because assessing data() outside of range [data, data + size) is undefined: It's no more dangerous than any other indexing operation. fread will certainly not access memory beyond the end of the array. It is up to you to make sure that func won't either. -- Remove del for email |
| Jorgen Grahn <grahn+nntp@snipabacken.se>: Jan 22 06:44AM > fread(buf.data(), buf.size(), 1, fp); > func(buf.data()); // consuming it > ``` Is this your actual use case -- to read a 64 gigabyte file into memory in one big chunk, then process the data? > I like vector, but the resize will initialize the whole vector as zero. > Which is unwanted, I don't even need it, it's a waste! I can see why you don't want it in this particular case, but normally it's a very good thing: std::vector<T> should help protect the invariant of T. If you want the huge file in memory as-is, one alternative is mmap(). It's a Unix function, but Windows has something similar and I suspect Boost has a portable API for it. Another option is to read the file in smaller chunks and process the data as it arrives, discarding it when you don't need it anymore. But that may not be possible depending on the file format and the task. /Jorgen -- // Jorgen Grahn <grahn@ Oo o. . . \X/ snipabacken.se> O o . |
| Paavo Helde <myfirstname@osa.pri.ee>: Jan 22 09:23AM +0200 > buf.resize(64 * 1024 * 1024 * 1024); > fread(buf.data(), buf.size(), 1, fp); > func(buf.data()); // consuming it Wow, 64 GB? If this is really a case, you should use memory mapping instead of reading it in in one go (but memory mapping is non-standard and has its own caveats). > ``` > I like vector, but the resize will initialize the whole vector as zero. > Which is unwanted, I don't even need it, it's a waste! Programming is an engineering disciple, which means you have to be pragmatic. As the file reading will most probably be several orders of magnitudes slower than the memory zeroing, it is a safe bet one can just ignore the time spent on zeroing in this scenario. Of course you can measure the timings first and then decide if this is something to spend time on or not. > How could I skip this? Buffer zeroing may cause significant overhead in other scenarios (not related to file reading). I ended up with making my own small wrapper class wrapping (the equivalent of) malloc() directly. But I did this only after repeated performance profiling had convinced me that one could win more than 10% this way, and there seemed to be no other way to squeeze the performance. TLDR; first profile your program and then decide which parts of it need speeding up. |
| Juha Nieminen <nospam@thanks.invalid>: Jan 22 08:12AM > I like vector, but the resize will initialize the whole vector as zero. > Which is unwanted, I don't even need it, it's a waste! As others have pointed out, in your particular scenario allocating a 64GB block of memory might not be really what you want (mmap is usually what's used to handle huge raw files as efficiently as possible, even though it's not a standard function). But anyway, to answer your actual question: std::vector isn't really designed to manage uninitalized memory, and this is by design. Perhaps the easiest way to get a block of uninitalized memory, C++ style, is this: std::unique_ptr<unsigned char[]> data ((unsigned char*)::operator new[](1234)); --- news://freenews.netfront.net/ - complaints: news@netfront.net --- |
| David Brown <david.brown@hesbynett.no>: Jan 22 09:36AM +0100 On 22/01/2019 07:30, Barry Schwarz wrote: > you know exactly how much data was actually read in? > Just out of curiosity, how long will it take your system to read a > 64GB block? My guess is that the OP is looking at this as an easy way to read in the whole file as simply as possible. Allocate a memory chunk that is bigger than any file he might want, but don't access that memory so that it is left unmapped (on many systems). Read the file into the buffer - no need to worry about buffer overrun with such a large buffer, and fread a larger size than any file might be. That way you are sure you've got everything in the file, with just a few lines of code. No need to read in chunks, or move or copy memory around. This is, of course, a /highly/ questionable approach in many ways. So my advice is for the OP to take a step back and look at what he wants to achieve - ask about that, rather than details of an attempted implementation. mmap() is probably a better answer, but that might depend on the OS, the types of file, etc. Or question the choice of C++ in the first place, especially if you are new to the language - make sure you use the right language for the job. In Python, this is: data = file("xxx.dat").read() But if you /really/ want to get a buffer of 64 GB space without touching any of it, use malloc(), not a C++ container. Just wrap it in an RAII class to be sure it is freed. |
| Paavo Helde <myfirstname@osa.pri.ee>: Jan 22 10:46AM +0200 On 22.01.2019 9:23, Paavo Helde wrote: >> ```cpp >> std::vector<char> buf; >> buf.resize(64 * 1024 * 1024 * 1024); Side remark: as written, this line is UB because of signed integer overflow, on most (all?) platforms. |
| leigh.v.johnston@googlemail.com: Jan 22 01:27AM -0800 > I like vector, but the resize will initialize the whole vector as zero. > Which is unwanted, I don't even need it, it's a waste! > How could I skip this? Simply use a dynamic array instead: char* buf = new char[64 * 1024 * 1024 * 1024]; Allocating uninitialised buffers (including allocating space for placement new) are about the only use-cases for dynamic arrays these days. /Leigh |
| "Alf P. Steinbach" <alf.p.steinbach+usenet@gmail.com>: Jan 22 11:10AM +0100 On 22.01.2019 09:46, Paavo Helde wrote: >>> buf.resize(64 * 1024 * 1024 * 1024); > Side remark: as written, this line is UB because of signed integer > overflow, on most (all?) platforms. Platforms with 64-bit `int` include the HAL Computer Systems port of Solaris to the SPARC64 and Classic UNICOS, according to Wikipedia's article on the issue. Cheers!, - Alf |
| "Öö Tiib" <ootiib@hot.ee>: Jan 22 02:19AM -0800 > ```cpp > std::vector<char> buf; > buf.reserve(64 * 1024 * 1024 * 1024); That is buf.reserve(0); with gcc 8.1.0 also following warning: warning: integer overflow in expression of type 'int' results in '0' [-Woverflow] > ``` > then what's the proper/elegant way? > I need pretty big buffer so I have to dynamically allocate it. Elegant way is to make code that compiles and runs and does what you actually want it to do then post it here as whole. It does not matter how fast our customers get incorrect results. Let's look at performance. DRAM is typically about order of magnitude faster than SSD so bottle-neck in your posted code is reading the file not zeroing memory. Even if you buy very decent SSD then freading 64 GB will take over 20 seconds no matter if the buffer was initialized or not. If you wan't better performance then you should make your algorithm less naive and to make it to work with sub-sets of whole data and to load it on need basis. Full HD video is about 3 GB/hour. |
| fir <profesor.fir@gmail.com>: Jan 22 02:23AM -0800 W dniu wtorek, 22 stycznia 2019 08:23:23 UTC+1 użytkownik Paavo Helde napisał: > As the file reading will most probably be several orders of > magnitudes slower than the memory zeroing, on good pc zeroing 64GB should take 6 seconds (in one core, if you use 6 maybe it could take 1 second - if they hae independand bandwidth, probably yes) reading from disk: some write good SSD hae 500 GB/s but i never tested it and i belive it may be not exact (to high) but lets take it then - 2 minutes |
| "Öö Tiib" <ootiib@hot.ee>: Jan 22 02:23AM -0800 On Tuesday, 22 January 2019 12:10:15 UTC+2, Alf P. Steinbach wrote: > Platforms with 64-bit `int` include the HAL Computer Systems port of > Solaris to the SPARC64 and Classic UNICOS, according to Wikipedia's > article on the issue. Yes, but Anhong won't get anywhere near one of such platforms anytime soon. |
| "Öö Tiib" <ootiib@hot.ee>: Jan 22 02:25AM -0800 On Tuesday, 22 January 2019 12:10:15 UTC+2, Alf P. Steinbach wrote: > Platforms with 64-bit `int` include the HAL Computer Systems port of > Solaris to the SPARC64 and Classic UNICOS, according to Wikipedia's > article on the issue. Yes, but Anhong won't get anywhere near one of such platforms anytime soon. |
| Bonita Montero <Bonita.Montero@gmail.com>: Jan 22 11:28AM +0100 > Wow, 64 GB? If this is really a case, you should use memory mapping > instead of reading it in in one go (but memory mapping is non-standard > and has its own caveats). Memory-mapping is slower and / or (depending on the platform) consumes more cpu-resources because there is usually no prefetching an faults on every page (thus the higher cpu-overhead). > Buffer zeroing may cause significant overhead in other scenarios (not > related to file reading). Is a buffer of std::byte really zeroed? I think std::vector simply calls the default-constructor, which does nothing for std::byte and thereby the whole internal initialization loop when enlarging the vector shoud be optimized away. |
| "Alf P. Steinbach" <alf.p.steinbach+usenet@gmail.com>: Jan 22 11:28AM +0100 On 22.01.2019 09:12, Juha Nieminen wrote: > is this: > std::unique_ptr<unsigned char[]> data > ((unsigned char*)::operator new[](1234)); I would not do that, because `std::unique_ptr` for an array effectively uses an ordinary `delete[]`-expression to deallocate, while the above requires a direct call of `::operator delete[]`. There is a difference: a `new[]`-expression may ask `operator new[]` for more than the size of the array, in order to store a count of objects (so that destructors can be called on the appropriate number of items), and the `delete[]` expression that `unique_ptr` uses by default, takes that into account, i.e. it may perform a pointer adjustment. As I recall it's unspecified whether this happens for an array of POD type. And then it /can/ happen, i.e. the code wouldn't be portable. A simple remedy is to just use an ordinary `new[]`-expression, or `make_unique`. Cheers!, - Alf |
| fir <profesor.fir@gmail.com>: Jan 22 02:30AM -0800 W dniu wtorek, 22 stycznia 2019 11:19:22 UTC+1 użytkownik Öö Tiib napisał: > you should make your algorithm less naive and to make it > to work with sub-sets of whole data and to load it on need > basis. Full HD video is about 3 GB/hour. 3GB/s SSD ? |
| "Öö Tiib" <ootiib@hot.ee>: Jan 22 02:36AM -0800 On Tuesday, 22 January 2019 12:28:32 UTC+2, Bonita Montero wrote: > the default-constructor, which does nothing for std::byte and thereby > the whole internal initialization loop when enlarging the vector shoud > be optimized away. No vector::resize must value-initialize the buffer if value wasn't provided. With char it means zeroes. |
| "Öö Tiib" <ootiib@hot.ee>: Jan 22 02:41AM -0800 On Tuesday, 22 January 2019 12:30:47 UTC+2, fir wrote: > > to work with sub-sets of whole data and to load it on need > > basis. Full HD video is about 3 GB/hour. > 3GB/s SSD ? Yes. 64/3 > 20 |
| Bonita Montero <Bonita.Montero@gmail.com>: Jan 22 11:46AM +0100 >> be optimized away. > No vector::resize must value-initialize the buffer if value wasn't provided. > With char it means zeroes. The container doesnt know anything about zeroes; it calls the default-constructor, which is a NOP in case of std::byte. |
| fir <profesor.fir@gmail.com>: Jan 22 02:54AM -0800 W dniu wtorek, 22 stycznia 2019 11:42:00 UTC+1 użytkownik Öö Tiib napisał: > > > basis. Full HD video is about 3 GB/hour. > > 3GB/s SSD ? > Yes. 64/3 > 20 but where you seen that fast ones? i not checked like some 2 years but it used to be 500 MB/s |
| fir <profesor.fir@gmail.com>: Jan 22 03:00AM -0800 W dniu wtorek, 22 stycznia 2019 11:54:53 UTC+1 użytkownik fir napisał: > > Yes. 64/3 > 20 > but where you seen that fast ones? > i not checked like some 2 years but it used to be 500 MB/s ok indeed i found some https://www.techradar.com/reviews/wd-black-sn750-nvme-ssd interesting also is how much speed drop down on non-sequential highly random read (0.5MB its disaster) |
| "Öö Tiib" <ootiib@hot.ee>: Jan 22 01:18AM -0800 On Monday, 21 January 2019 23:18:51 UTC+2, Jorgen Grahn wrote: > management[0] and hadn't wrapped it properly. But often you also want > to run dynamic checkers on others' code to find out how ill-behaved it > is; then leak checking is a useful feature IMO. Yes, sometimes libraries contain defects, also standard libraries, also leaks. So always support exact versions of dependencies and update those only for reason and as separate pull request. Again major headache avoided close to effortlessly. Leaks result with symptom that memory usage grows over time. Sometimes the leaks are "logical". For example more objects are put into some global container than erased from it over time. Such leaks can't be detected with any tool. On worst cases it can be quite challenging to figure it out. > Address, leak and undefined, then. I see -fsanitize=thread too. > The GCC manual says some of the sanitizers cannot be used together, > but doesn't say which ones. If you have continuous integration then there are no much difference if it runs tests 2 times or 10 times (with different compilers and combinations of instrumentation). Computing power is dirt cheap so people waste it on total trash (like bitcoins). So I quite strongly suggest to do that. Little benefits add up into greater perceived quality at low cost. > > These will typically detect if code misuses operators of containers > > or iterators; ASan will detect if code misuses pointer arithmetics. > Thanks; I (and others) tend to forget that option. Yes, the assertions of library are way easier to comprehend than same issue raised by ASan for example. Library asserts first so these can be used as combination, too. |
| Josef Moellers <josef.moellers@invalid.invalid>: Jan 22 08:42AM +0100 On 21.01.19 17:45, Scott Lurndal wrote: >> daysSinceModification or fileAgeInDays would be better". >> (S)He could have made this more clear, though. > struct stat's st_atime, st_ctime, st_mtime seem clear enough to me... ??? I was referring to page 5. There the author has a bullet "No lower-case L or upper-case o, ever." and to the right of this four closely spaced variable/field declarations, the first of which has a lower-case l. At first glance, I had taken this as a counter-example, as the lower-case l really resembles the upper-case I. But looking more closely at this page, it really looks like the author wants to replace the "int d" by one of the four declarations to the right of the arrow the first one of them declares the variable/field with the lower-case l! If the author ever reads this: pdf pages are virtually free, so are PowePoint or Libre Office Presenter slides! Please don't try to cram as much as possible onto a single page, it confuses more than it helps. Josef |
| ram@zedat.fu-berlin.de (Stefan Ram): Jan 22 06:15AM >I like vector, but the resize will initialize the whole vector as zero. >Which is unwanted, I don't even need it, it's a waste! Maybe you could use a #include <vector> #include <cstddef> .. ::std::vector< buffer_byte >v( buffer_size ); with struct buffer_byte { buffer_byte(){} ::std::byte x; }; ? The constructor will be called for each buffer_byte, but it will not do anything. Also make sure buffer_size fits into ::std::vector::size_type. if( !v.empty() ) { byte * beg = &v.front().value; byte * end = beg + v.size(); ~~~ } . Other possibilities include: Using a vector with a custom allocator whose "construct" does nothing. |
| You received this digest because you're subscribed to updates for this group. You can change your settings on the group membership page. To unsubscribe from this group and stop receiving emails from it send an email to comp.lang.c+++unsubscribe@googlegroups.com. |
No comments:
Post a Comment