For memory, what should be done when you need to constantly grow a vector to an unknown upper limit?

Suppose that you are dealing with a potentially infinite amount of data. Suppose further that you do not have this data stored in memory, but can generate individual terms at will. Finally, suppose that you want to do some experiment on this data that will involve checking a large but unknown amount of terms in a way that necessitates keeping a great many of them in memory. Toy problems with Recamán’s sequence, like "find the minimum number terms needed in that sequence for the first 25 even numbers to have appeared", are what I have in mind as typical examples.

The obvious solution to this sort of problem would be to write some code like:

list<-c(first term)
while([not found enough terms yet])
  if(this term worked){list<-c(list,nextTerm)}

However, building a big vector like this by adding one new term at a time is your memory’s worst nightmare. The alternative that I often see suggested is to pre-allocate a big vector in memory by making the first line of your code something like list<-numeric(10^6), but those solutions suppose that we have some rough idea of how many terms we need to check, which isn’t always the case. So what can we do when we are dealing with an ever-growing list of unknown required length?

Stack Overflow Asked on November 15, 2021

1 Answers

One Answer

This is very popular subject in R check this answer: Summing up: Do not use c() to bind as providing value by index [ is much faster. I know that it might seem surprising that you could grow pre-allocated vector. Make an iter variable before while loop and increase the index inside the if statement.

Normally like in Python you do not have to care about it when using append. Even starting with empty list is not an problem as the list (reserved memory) grows expotentialy (x2x2x1.5x1.2...) when you pass some perimeter number of elements. Link Over-allocating

Answered by polkas on November 15, 2021

Add your own answers!

Related Questions

If and excel formula for matching

1  Asked on November 23, 2020 by howaboutno


Pyspark Transpose

1  Asked on November 23, 2020 by sawan-s


How to aggregate data, pass and fail as final result?

1  Asked on November 23, 2020 by user584018


Comparing two lists for same value

1  Asked on November 22, 2020 by sebaku


Crystal Reports won’t display related table data

2  Asked on November 22, 2020 by rharris


Upgrading spring cloud version from Edgware to Hoxton

3  Asked on November 21, 2020 by ruchira-gayan-ranaweera


How to prevent button form Angular?

5  Asked on November 20, 2020 by user13976004


SQL union seems to fail for me

1  Asked on November 19, 2020 by sasha-peric


My Projectiles Stop Shooting Problem How To Fix?

1  Asked on November 19, 2020 by habib-ismail


Why Linq Distinct() does not work for my ConsultantList?

1  Asked on November 19, 2020 by karim-husein


Flutter app: simple way to remember a user to do something

1  Asked on November 19, 2020 by mactrix


Loop through a new json object every X seconds

2  Asked on November 19, 2020 by unluckylad


why css file was damaged on server

2  Asked on November 18, 2020 by hamid-talebi


Ask a Question

Get help from others!

© 2021 All rights reserved.