Police in Malaysia say nearly 20 people, including 13 Japanese, were detained over a suspected scam operation that apparently targeted Japan.