nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Child Item

I bet you didn't know that this thread would have a part 2

IamMetaldave

BaZmO* wrote: »

This post is the 10,000th post

Pipped.

Mod: Continuation from the old thread, which is here.

Find more posts tagged with

Quick Links

Comments

764dak

Will Smith technically doesn't play himself on Fresh Prince. His name is Willard Smith while his character's name is William Smith.

New Home

I heard someone say yesterday that the combined data collected for the photo of the black hole was too big to be transmitted by internet, and was physically brought to one place to be collated.

Too big for the internet!

Fourier

New Home wrote: »

I heard someone say yesterday that the combined data collected for the photo of the black hole was too big to be transmitted by internet, and was physically brought to one place to be collated.

Too big for the internet!

I'm in a email conversation with somebody now who knows a lot more about the experimental and observational details than I do, hopefully I'll have a nice summary soon, it's just so much info about stuff new to me! The paper itself wasn't enough to fully get all the details.

However it wouldn't surprise me, there were similar issues in the LHC solved by dumping about 99% of the data with selection algorithms. Although even the data remaining allowed a huge certainty on the detection of the Higgs boson.

Professor Moriarty

Cymothoa exigua is a louse that enters a fish through it's gills. About a centimetre in length, the male lives in the fish's gills while the female enters the fish's mouth and attaches itself to the tongue while bleeding the tongue dry. The tongue falls off and the louse then attaches itself to the stub and becomes the fish's tongue. Usually, this seems to have no obvious negative effect on the fish.

Ahorseofaman

Fourier wrote: »

hopefully I'll have a nice summary soon,

Looking forward to being both enlightened and confused,really enjoy your contributions.(not that I can grasp a lot of them):cool:

Fourier

Well there are five release papers thus far (there is to be a sixth). I've just read them and had very helpful pointers from an old colleague.

So how was the black hole image obtained?

A set of eight telescopes around the globe focused on the M87 galaxy's core on April 5, 6, 10, and 11 in 2017. This galaxy was chosen instead of closer ones like our own because it is very active/bright without being too far away and the central black hole was predicted to be of a very large size. In addition the central black hole was predicted to be relatively static. What this means is that unlike our own it doesn't yo-yo up and down in terms of brightness.
The light that was collected is synchotron radiation. Basically electrically charged particles throw off radiation as they are swung around the black hole. The radiation is way outside the human visual spectrum, so it's not a true colour image. These charged particles are all part of a highly magnetic stream of fluid coming from stars torn apart by the black hole.
Once the data was collected a bunch of common errors have to be filtered out. These are things like slight errors from the warping of the disks in the telescopic arrays, parts of the array being slightly unfocused, etc. The telescope in Mexico known as the LMT had a bit more of these than others as the galaxy was only accessible to it during the evening when the disks warp slightly at the millimeter scale from changes in temperature going from day to night and it is harder to focus in evening light.
All these errors have to be filtered out, but fortunately they are well understood so can be isolated easily. Each station has built up over years of study a statistical catalogue of how likely parts of their array are to be defocused or be slightly hot etc. Again the LMT in Mexico was the hardest to do this for because it's the youngest.
There's also well understood things like removing effects from the Earth's atmosphere and intervening stellar dust. In addition the slight differences between each telescope due to their different positions on the Earth
So after this we have the observational data with errors removed. All the data from all eight stations were collected, effectively giving you something like data from a telescope the size of the Earth. Now to reconstruct the image from that data.
There is an old piece of software called CLEAN used in Radio Astronomy that takes the amount of light obtained in each detector on the telescope, angles and distances between detectors and uses this information to build an image. However in this case half the teams used CLEAN and the others a newer algorithm called RML.
However these algorithms have a lot of settings. Basically they need to know what data is considered part of the black hole image (there will be data from other sources) what parts of the image to use as anchor points to detect warping in the picture and several others I don't understand. The details of imaging software is above my paygrade.
To determine what the settings should be they tested the algorithms against generated data. They basically simulated black holes and a few other astronomical objects, about 10,000 in total. With the "true" images from the simulations they fed the algorithms a sample of the simulation data to see if it could correctly rebuild the simulated image. And they did this for all 10,000 until they were satisfied.
Three teams did this coming up with very similar settings which were then given to the teams using CLEAN and RML. In the end every team obtained an image and the average was used to obtain the final image.
This gave an image for each of the four observation days.
The final step involved General Relativity itself. The images were fed into Einstein's field equations which can tell you how magnetic fluids like the shredded stars should move near a black hole. The equations basically predicted "If a black hole looked like this on April 5th it should look like this on April 6th". The image output by the equations was an extremely close match for the actual April 6th photo. This was done for every day and compared against others (e.g. using the equations on the April 10th data and predicting what April 5th must have been like). Also each image was checked against a simulation of a black hole of that size and spin using General Relativity.
This confirmed that these indeed seem to be images of a rotating black hole behaving exactly as General Relativity says it should.

Ipso

New Home wrote: »

I heard someone say yesterday that the combined data collected for the photo of the black hole was too big to be transmitted by internet, and was physically brought to one place to be collated.

Too big for the internet!

Imagine if someone quoted it!

Buford T. Justice XIX

Ipso wrote: »

Imagine if someone quoted it!

Forum ban, I'd expect:D

StupidLikeAFox

New Home wrote: »

I heard someone say yesterday that the combined data collected for the photo of the black hole was too big to be transmitted by internet, and was physically brought to one place to be collated.

Too big for the internet!

5 petabytes apparently, here is one of the scientist with all the storage:

New Home

Ipso wrote: »

Imagine if someone quoted it!

Buford T. Justice V wrote: »

Forum ban, I'd expect:D

The hamsters wouldn't stand a chance, and the black hole would absorb everything, including your own computer.

Capt'n Midnight

The Israeli moon lander probably isn't doing so well.

Last photo at 20Km above the surface.

That's about the same as 2 miles on earth, excluding air resistance.

If only they hadn't strayed too close to the secret Nazi moon base.

sunnysoutheast

New Home wrote: »

I heard someone say yesterday that the combined data collected for the photo of the black hole was too big to be transmitted by internet, and was physically brought to one place to be collated.

Too big for the internet!

It's very common practice to move very large data sets into the cloud via physical means rather than over the internet.

AWS will send you out ruggedized drives if you ask for one, or even a shipping container!

mrsoundie

New Home wrote: »

I heard someone say yesterday that the combined data collected for the photo of the black hole was too big to be transmitted by internet, and was physically brought to one place to be collated.

Too big for the internet!

Just had a look at this link and it says a time of 521 Days 5 Hours 59 Minutes 56.27 Seconds over a 1 Gig connection.

Capt'n Midnight

New Home wrote: »

I heard someone say yesterday that the combined data collected for the photo of the black hole was too big to be transmitted by internet, and was physically brought to one place to be collated.

Too big for the internet!

Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.
- Andrew S. Tanenbaum

dasdog

New Home wrote: »

I heard someone say yesterday that the combined data collected for the photo of the black hole was too big to be transmitted by internet, and was physically brought to one place to be collated.

Too big for the internet!

One keyword here is transmitted. ISP's routers are configured with much higher priority to download, which is what most end users consume, to upload. If you run a speed test on your broadband connection there will probably be a ration of 6:1 or more favouring download.

From my limited networking knowledge it's doable but the data would need to be transferred with error correction (as the result set could be compromised) and over secure tunnels (as it would be open to sabotage) which would greatly slow the process down. Much faster to put the data on encrypted drives and securely ship them. It's a very low tech solution but it does occur when a company is moving a large amount of data from on premise/existing data centre to a cloud offering.

We sometimes get questions in work for an ETA of how long a data transfer would take using a certain solution, its really urgent etc as there is a customer go live in a couple of days. Its impossible to gauge these things but when the answer is in weeks their emotions usually go from panic to realisation to offering up the money for a solution they should have originally budgeted and planned for.

Capt'n Midnight

SpaceX Falcon Heavy is due to take off in about 5 minutes.
It's a big rocket thing.

https://www.youtube.com/watch?v=ojvu2u28CIY

Franz Von Peppercorn

Ahorseofaman wrote: »

Looking forward to being both enlightened and confused,really enjoy your contributions.(not that I can grasp a lot of them):cool:

I just thank his stuff and nod sagely to myself.

Franz Von Peppercorn

sunnysoutheast wrote: »

It's very common practice to move very large data sets into the cloud via physical means rather than over the internet.

AWS will send you out ruggedized drives if you ask for one, or even a shipping container!

That’s right. Road is faster when you get to the multi terabyte level of data.

https://www.spiria.com/en/blog/big-data/100-petabytes-on-the-road-the-snowmobile-solution/

That blog talks about transferring huge amounts of already collected data.

Take for example DigitalGlobe, an American company specializing in satellite imagery. Over its 17 years of operations, it has gathered over 100 petabytes of images of our planet’s surface. Every year, its constellation of commercial satellites gathers 10 petabytes more data. Until recently, DigitalGlobe archived its images on tapes and sent out orders to clients in FTP format or on hard drives by courier--a cumbersome process that required several hours of handling.

The company decided to upload its enormous library on Amazon’s cloud to provide a faster, more competitive service to its customers. Therefore last year, a Snowmobile truck parked at DigitalGlobe’s headquarters in Colorado, transferring 54 million high-resolution images in just a few weeks and making the entire library of images available online. And, to keep the library always up-to-date, the 80 to 100 terabytes of new data produced every day are transferred online to Amazon S3 on a daily basis.

In the absence of this kind of transfer, as the article also explains uploading 50-petabytes is an operation that takes 28 years and 7 months on a 1,000-megabit per second fibre optic cable. My upload speed is 3Mb/s so it would take 950 years or so.

(Most businesses are SDSL not ADSL meaning the download and upload speed are the same and in general relatively high, but it makes no odds with this size of data).

ohnonotgmail

I wouldn't even like to think how much that amazon storage is costing per month

sbsquarepants

Practically everyone is aware of the stereotype of older people getting up at the crack of dawn and going to bed early in the evening whereas younger people tend to do the opposite - night owls who sleep till stupid o'clock in the day.

Well it turns out it might be an evolutionary roster system. Recent studies of a primitive hunter gatherer society in Tanzania (the Hadza) has revealed that in the absence of any time constraints or measurements, they tend to do the exact same thing. In fact over the course of the 3 week study of one group of 30odd people, there were only 18 minutes where the whole tribe were asleep simultaneously.

The researchers have surmised (reasonably enough I think) that it may well be natures way of having a sentinel on guard round the clock to keep an eye out, ward of predators and so on.

So remember, next time you're about to lay into your teenagers for sitting up all night, bingeing on netflix and internet porn - you weren't eaten by a lion where you...... so maybe a thank you is more appropriate

quickbeam

If Manhattan was as densely populated as Alaska there would be 25 people on the whole island.

Manhattan - 28K per km2
Alaska - 0.49 per km2

If the world was as densely populated as Manhattan we'd all fit in New Zealand.

partyguinness

Newborn babies open their eyes when placed in a dark room.

partyguinness

The guy in the white suit next to Lee Harvey Oswald when he was shot is still alive. He also survived Pearl Harbour.

https://en.wikipedia.org/wiki/Jim_Leavelle

KevRossi

partyguinness wrote: »

The guy in the white suit next to Lee Harvey Oswald when he was shot is still alive. He also survived Pearl Harbour.

https://en.wikipedia.org/wiki/Jim_Leavelle

Now that is the exact thing that I constantly come on this thread for!

Chapeau Monsieur!

Chapeau!

mzungu

The Empire State building took a little over a year (410 days) to be built. It was finished 12 days ahead of schedule.

lan

dasdog wrote: »

From my limited networking knowledge it's doable but the data would need to be transferred with error correction (as the result set could be compromised) and over secure tunnels (as it would be open to sabotage) which would greatly slow the process down. Much faster to put the data on encrypted drives and securely ship them.

Error detection and retransmission is built into TCP/IP, the most common transport protocol on the internet. It doesn’t add much overhead (unless your connection is unreliable).

Encryption also doesn’t increase the size of data, and can be done effectively in real-time, far faster than the data can be transmitted across the net.

For datasets that large, it’s really just a matter and bandwidth, and as you’ve said, it’s much faster to just send the disks.

dasdog

lan wrote: »

Error detection and retransmission is built into TCP/IP, the most common transport protocol on the internet. It doesn’t add much overhead (unless your connection is unreliable).

Encryption also doesn’t increase the size of data, and can be done effectively in real-time, far faster than the data can be transmitted across the net.

For datasets that large, it’s really just a matter and bandwidth, and as you’ve said, it’s much faster to just send the disks.

I was inferring UDP and I'd love to know how securing a tunnel has no overhead.

ytpe2r5bxkn0c1

If you open your eyes in a pitch-black room, the colour you see is called 'eigengrau.'

KathleenGrant

Nevaeh Shaggy Destroyer wrote: »

If you open your eyes in a pitch-black room, the colour you see is called 'eigengrau.'

Why is it not black? It looks black.

ytpe2r5bxkn0c1

KathleenGrant wrote: »

Why is it not black? It looks black.

It's a dark grey colour that's in perfect darkness, as a result of signals from the optic nerves.