Thursday, 9 June 2016

A small trial in your garden proves very little!

Don't you believe it.
My grandson Arthur Brook is a budding researcher
It is good to try out new ideas and how we do things in the garden. Without trial and error we would be stuck in a rut and never change.
What worries me is that sometimes gardeners construct 'little trials' and proclaim to the world they have found something new. Worse when they read dubious information that has no scientific validity - often promoted by advertising or someone with an axe to grind - they not only believe it but pass it on as a fact.
And where does that leave me who sets out to advise gardeners?

As a species we would have never have advanced without having been taught by others how to do things and to believe what we are told! It's this same inborn trust that leaves us very vulnerable to getting things wrong.

Although I am suggesting today that what you do in your garden should be scientifically valid, how do you know? There are many dubious 'scientific' facts and explanations peddled in horticulture. Even research as high as University level is sometimes flawed. All the more so when 'the publicity department' has got their hands on research information. Worse still, when research is interpreted by journalists who add their unproven world view as a thin veneer on what has been actually shown.

Nature provides its own trials – and tribulations
I have been involved with students who have done so called trials as part of their education. It is a good thing to have an investigative mind. I have never known them to actually prove anything!
There are too many variables. What seems to be a genuine result is often complete chance. To have scientific validity results need to be duplicated and statistically valid. Not only do experiments need a large number of repetitions in their make up, the actual experiments need to be repeatable by other investigators. How else does one avoid wishful thinking, bias, and very human error?

As an example suppose a gardener is testing a new product, what can go wrong?
If the actual product is used by millions of gardeners and perhaps used by a whole industry - perhaps a high nitrogen fertiliser - you will get a result in as much as the plant may grow quicker and turn a bright green! (But that won't tell you whether it is a good thing or a bad one).
On the other hand if that product is something new and the effects are small the very best you can show is it might work for you.
So many things can go wrong in a trial and the possible errors are legion!
Is there a control? I mean is there really a comparison where two test treatments are exactly the same other than the one variable being tested?
I mean exactly the same! Is the soil for the duplicated treatments identical? Is the drainage uniform? Are light levels the same? In a small garden illumination is very variable. What about the test plants - did they really start out exactly the same? They were of course the same variety!  Is the watering the same?
Are there enough repetitions in the project? Maybe a dozen might give you a clue if the differences are dramatic. If the differences are small you will need many more.
What about the vagaries of your pest and disease? You might eliminate slug damage from your analysis by discarding that example from your results, but is that not bias, slugs might like whatever you are testing? Worse do you eliminate the slug damage if doing so favours the result you prefer but you fail to make such an adjustment the other way round!

Your slug damage might be a black swan

You might inadvertently demonstrate something different to what you mean to. Suppose the said product really was quite useless and you had decided to work it into the ground. You would need similarly to disturb the ground in your control otherwise you would show the effect of hoeing!
Do you have an interest invested in the result? To win an argument, prove a philosophy or perhaps just to be nice to whoever gave you the product to test?
Even if you get a result for your conditions, in that season, in that weather, with that unexpected frosty night, for that time you were unable to water when you got back late, for those conditions you subconsciously thought your product would do well in when you set up the experiment, on your own soil with your own favourite variety: you cannot claim to the world you have proved anything at all.
Then how do we learn?

It gives me a lot of fun to bury newspaper. Eric Robson once filmed me burying newspaper for a TV programme!
By all means try out things different and see how it works for you. Perhaps do little tests. When I started to think about burying newspaper I checked how much water an inch of the Times newspaper would absorb! My result showed it to be substantial but I made the mistake of thinking ten times that thickness would hold ten times the water. It does not as water in deep profiles drains away more.
When my friend Peter read that the Himalayan balsam spreads when the seed floats down rivers and streams he checked out his doubts by adding the seed to a bucket of water and finding it sinks! He would of course agree that like silt, seed might be washed down the river but it is likely that most of the spread is by the well known mechanical dispersal

Himalayan Balsam Impatiens glandulifera
Mechanical dispersal does the job very well
Both the above are petty examples but do illustrate you should have a questioning mind.

The best way to learn is to be shown and taught by skilled experienced practitioners or to read books and magazines. It is also the worst way when there is so much myth and magic that comes from such sources. Gardeners who have science in their background do have an advantage when their knowledge of nature enables them to ask the right questions and sort the wheat from the chaff.
When I am out of my own area of expertise I find myself learning by listening to those I respect and following their lead to check things out further. In matters medical for the last ten years I have followed the sound common sense, independence, experience and encyclopedic knowledge of worldwide medical research by Dr. David Grimes.
When I speculate about hybridity, I freely confess that without access to the writing of geneticist Gene McCarthy I would not have been able to put pen to paper.
When it comes to gardening I wend my own way.

There are many ways to garden. Fortunately most of them work even if methods are not perfect or are even disputed. Just do it and enjoy it. Be open to new ideas and be prepared to make mistakes and learn. Soak up information wherever it comes from but always question.
Every year I try things different but I never have done a designed 'trial'. I have had my triumphs and disasters. I hope I will have many more.

I bumble along

I am trying my home made charcoal as a compost for my tomatoes this year
I hope in my zeal to question the validity of a trial that this not deter you from experimenting with things new. Suppose you are considering trying a new variety of vegetable. You might try the new one but still take a crop of the old one as insurance! The new one might prove to be much nicer but perhaps harder to grow. You will make a subjective decision about what you do in the following year!

Not much of a trial
I took blogger Mark Willis’s advice last year  and tried climbing french bean ‘Cobra’ - in my own case to grow it for an early crop in my greenhouse.
(The previous season I had done the same with runner bean Polestar. It grew and yielded very well. Unfortunately most of the beans matured at the same time and Brenda does not really like runner beans. I had chosen Polestar as a self fertile runner bean as commercial research shows in poly-tunnels tunnels there are insufficient pollinators for cross pollinated varieties. When my leaky greenhouse positively buzzed with bumble bees I realized I had not really needed to chose Polestar)

Last year I sowed six seeds of Cobra in a three inch pot in my warm conservatory on the first of April – it is my only heated growing environment and as we live in that room most of the time I am only allowed a very small corner. The germinated seedlings were planted together at a single station about mid April in my unheated greenhouse. They sulked for a while and the slugs had a go at them. When warmer weather arrived they took off to give a dozen large boilings for the two of us before the outdoor beans were ready. Cobra is a wonderful bean and it grew very well.

This year the equivalent young beans are still sulking. Will they survive until warm weather comes? It’s all trial and error!
I write these posts in advance. In this case the beans were sulking too long and I sowed some more. Now in warmer weather they are chasing each other roof wise!

The beans on the right were sown nearly a month later! In a different season the first sowing might have raced away and in another would be dead!
I never do the same things twice! My indoor sugar peas in the higher picture were eaten by the mice and the re-sowings this time in a seed tray are just starting cropping

More on commercial research
I started this post by saying that any attempt to do real research and to do a genuine trial by gardeners such as me is a complete waste of time as the results will never be statistically valid and will always be seriously flawed rendering them worthless. I wonder how much we should trust ‘real research’.

Perhaps not very much! As I have hinted commercial research is not really geared to the very different needs of we amateurs. Commercial growing is a completely different ball game.

Horticultural research is not done for amateurs
We live in a society where researchers are paid by results. We live in a world where vested interests want to persuade you. We hear from the media much conflicting information based on so called research.
Trials can be designed in such a way as to give almost predictable conclusions that will serve a vested interest. Researchers are not required to publish negative results they don’t like. So called ‘black swans’ are rejected. Neutral conclusions go unreported because they are not very sexy. Worse some research receives world wide publicity and yet is based on very few replications and has never been reproduced by a disinterested party.

The kind of horticultural research about which I am particularly doubting are those trials that are exclusively laboratory based. I am particularly dubious when chemicals are directly applied to a plant or an animal when testing for safety in use. Things need to be verified in real field situations.
I am also suspicious where genome studies on plants are used predictively without really growing them.
I asked scientist friend Peter which research I should respect. He replied that I should enquire who is paying.

One final irony. The most dubious claims for new ‘wonder products’ are made without any research at all. The actual facts get in the way
David Grimes in full flood
Gene McCarthy writes about how new hybrids stabilise
I bury newspaper
My own trial lacked credibility
My post about black swans had some of Cathi’s very nice pictures


  1. “Every year I try things different, I have had my triumphs and disasters. I hope I will have many more, I bumble along.”

    Well, that’s how I would have described myself as a gardener – just trial and error, I have no formal gardening qualification and never taken a gardening course in my life so I wouldn’t dream of comparing myself to all those who actually have a horticultural background – but I am curious and I ask questions and I am grateful every time more knowledgeable people take the time to answer, including you :-)
    Thanks for another great article.

    1. Thank you for your nice comments Helene.
      Although I have been involved in education all my life and cannot resist 'passing on my knowledge' I am a great believer in self learning - especially with all that easily accessible information out there (shame some of it is wrong).
      I think the best way to learn is to have a passion and follow it

  2. I'm glad you liked the "Cobra" beans, Roger. I have "trialled" (!) them many times and the results have been fairly consistent - but never exactly the same, since as you point out, there are so many variables. When growing plants on a small scale, I think a lot of luck is involved. I also agree that the internet is responsible for perpetuating (and often inventing) many myths. There is no substitute for practical experience when it comes to gardening.

    1. One's own practical experience can create one's own myths, of course. I used to work in an office where a standard light only turned on the first time I used it only after I'd tipped it sideways. It was several days later, after carefully tipping it each time I turned it on, that I realised it merely took several seconds to light up after being switched on and that tipping it had merely filled in the time between switching and lighting. Didn't half feel silly!

      I have wondered about the accuracy of citizen science. I asked once about the problem of not knowing whether everyone taking part was doing the experiments right and was told that sheer weight of numbers made the results accurate.

    2. Never feel silly about what you do in the garden Helen!
      I attended a wonderful lecture at the Castle Howard Yorkshire Arboretum (open to everyone) last night about tree pest and disease and heard how valuable citizen science was in reporting the spread of tree pest and disease.
      Apparently there is a great deal of ash dieback this year

    3. At least with our veg Mark we can usually have a plan b) or even c)!

  3. It is very difficult to do what when teaching we called a fair test. There are so many different variables over which you have no control or which are very difficult to keep the samle. Then there is the variable that one person's patch of land is ivery different to that of another person. Also you don't have to travel far in your own garden for the ecology and microclimates to be very different. Alternatively if things are tested in laboratory conditions the findings may not replicate in the natural environment where other variables may affect the outcome. For this reason it is very difficult to recommend that other gardeners behave in certain ways or grow certain things. We have grown things in exactly the same way two years in succession with very different results. We grew Cobra for the first time last year and it grew brilliantly from the start. This year the plants are probably the worst of our runner beans and climbing French beans. You can't win really win can you?

  4. Lots of thoughtful stuff Sue.
    I think in teaching, experiments - usually with a certain result known to the teacher - are very valuable.
    So called trials in teaching are usually staged demonstrations - and perhaps no worse for that.

  5. You are so singing the same tune as I believe. Home gardens are the worst at this. Growing plants zones beyond their own and proclaiming success when they only have installed the plants for not even one full year. They misrepresent how and if the plant will live or perform as advertised. Science is like you mentioned too. Tests are designed for desired results often. Money funded for such limitation too. Science not sharing with other related disciplines. "Trials can be designed in such a way as to give almost predictable conclusions that will serve a vested interest." Yes it is "follow the money". I will be visiting a native education gardens and they have done scientific study on navitars compared to the species native. They compared sugar production and nectar benefit and some of the results were surprising. But what they noted was these results are not going to be consistent in other areas around the country. That is a very fair observation. I will have a post on this after I visit this center since I often tell garden bloggers navitars are not always as beneficial as native species, so stop saying you have a native garden when in fact many have the navitars.

  6. You have given this a great deal of thought Donna - as you always do on your blog. Especially things novel or not clear cut - even controversial
    I suspect we are on opposite sides of the native plant debate
    My own view is that 'native' is a manmade concept and that other peoples's native plants have much to offer us - both to gardens and the wild.
    That is not to argue that certain pristine habitats should not be preserved.
    Which ever side we are on it would be nice to know that 'research into the facts' is done in an honest way - i don't mean people are consciously dishonest, its just that we all look for the facts that support our position!

  7. Very good post. I wonder if you are familiar with the plant evaluations done by the Chicago Botanic Garden, and what you might think of them.

  8. It looks very valuable as a resource for gardeners - done by a professional disinterested botanic garden, trials repeated on other sites and done over a long period.
    The RHS over here do valuable work of this kind

  9. Great stuff Roger, as you would expect I agree wholeheartedly with your views. There are so many variables influencing the growing of plants it is very difficult in many ways to be truly objective and I love Peter's comment about who is paying. I suppose one could say that the only time true results are obtained is when testing under laboratory conditions but these can only be taken as theoretical and as such can only be treated at best as guidelines when referred to in practice.

    1. The best research is done by disinterested parties


Related Posts Plugin for WordPress, Blogger...